This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 



BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the 
original documents submitted by the applicant. 

Defects in the images may include (but are not limited to): 

• BLACK BORDERS 

• TEXT CUT OFF AT TOP, BOTTOM OR SIDES 

• FADED TEXT 

• ILLEGIBLE TEXT 

• SKEWED/SLANTED IMAGES 

• COLORED PHOTOS 

• BLACK OR VERY BLACK AND WHITE DARK PHOTOS 

• GRAY SCALE DOCUMENTS 

IMAGES ARE BEST AVAILABLE COPY. 



As rescanning documents will not correct images, 
please do not report the images to the 
Image Problems Mailbox. 



This Page Blank (uspfo) 



(19) 



J 



(12) 



(43) Date of publication: 

19.05.1999 Bulletin 1999/20 



Europaisches Patentamt 
European Patent Office 
Off ice europeen des brevets (11) EP 0 917 371 A2 

EUROPEAN PATENT APPLICATION 

(51) Int. CI. 6 : H04N 7/26 



(21) Application number: 98121413.3 

(22) Date of filing: 11.11.1998 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES F1 FR GB GR IE IT LI LU 
MCNLPTSE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 11.11.1997 J P 308387/97 
28.1 0.1 998 JP 321 375/98 

(71) Applicant: 

Matsushita Electronics Corporation 
Kadoma-shi, Osaka 571-8501 (JP) 

(72) Inventors: 

• Matsuura, Shun 
Tokyo 166-0014 (JP) 

• Kageyu, Satoshi 

Fujisawa-shi, Kanagawa-ken 251-0002 (JP) 



• Kato, Hitoshi 
Tokyo 133-0065 (JP) 

• Asai, Kayoko 
Tokyo 125-0054 (JP) 

• Yoshio, Hiroaki 
Tokyo 124-0012 (JP) 

• Kato, Tsuyoshi 
Tokyo 144-0046 (JP) 

• Hayakawa, Yoshihiro 
Yokohama, 241-0823 (JP) 

• Maruya, Kensuke 
Tama-shi, Tokyo 206-0025 (JP) 

(74) Representative: 

Finsterwald, Manfred et al 
Patentanwatte, 

Manitz, Finsterwald & Partner GbR, 
Robert- Koch-Strasse 1 
80538 Munchen (DE) 



(54) Moving Image composing system 



(57) A moving image composing system for display- 
ing reduced and encoded images at arbitrary positions 
on a multi-image raster. The moving image composing 
system comprises a moving image reducing unit for 
reducing a moving Image inputted, a moving image 
encoding unit for encoding the reduced moving image 
to produce an encoded moving image, an accumulating 
unit for storing and accumulating the encoded image, 
and a moving image composing unit for combining 
more-than-one encoded moving images accumulated 
to produce a multi-image combined moving image. The 
moving image encoding unit performs the encoding 
processing to facilitate the change of positional informa- 
tion the reduced moving image has, in consideration 
with the case that the change of the position information 
occurs due to the disposition of the reduced moving 
image at its combination. When there is a need to 
change the positional information of the reduced mov- 
ing image due to The combination processing, xhe mov- 
ing image composing unit changes the positional 
information to positional information required for the 
combination processing. 
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Description 

BACKGROUND OF THE INVENTION 
[Field of the Invention] 

[0001] The present invention relates to a composing 
system or apparatus, in the case of conducting simulta- 
neous multi-image reproduction of a plurality of moving 
image (picture) data for one image or picture, using a 
method of encoding (coding) each of the moving image 
data and a composing method of producing a multi- 
image combined moving image corresponding to an 
arbitrary combination of the moving image data. 

[Description of the Prior Art] 

[0002] As exemplified by Japanese Unexamined Pat- 
ent Publication No. 7-298263, there has been known a 
method whereby a synthetic code string obtained in a 
manner that a frame of a moving image is divided into a 
plurality of areas and code strings each produced by the 
encoding of each of the areas are connected to each 
other becomes identical in syntax with a code string cre- 
ated by encoding the image without the division into the 
areas. 

[0003] Fig. 51 is a block diagram showing an arrange- 
ment accomplishing this method, and an operation 
thereof will be described hereinbelow with reference to 
this illustration. 

[0004] First of all, a moving image inputted is divided 
through an image divider 61 into a plurality of areas in 
units of frame. At this time, the image divider 61 deter- 
mines a division method on the basis of a image division 
control signal from a division control unit 75. In this 
instance, one frame is* vertically divided into three 
areas, and divided-image signals are respectively taken 
as 77, 78 and 79 in the order from the left-hand area of 
the original image, and the divided-image signals 77, 78 
and 79 are respectively introduced into encoders 62, 63 
and 64. Each of the encoders 62 to 64 receives a size of 
the divided frame and ah encoding condition from the 
division control unit 75 to encode the inputted divided- 
image signal, and then outputs as a code string 80, 81 
or 82 to a code string synthesizer 74. Subsequently, the 
division control unit 75 outputs a code string synthesis 
signal which informs the. code string synthesizer 74 of 
the frame division method according to the instruction 
given to the image divider 61, and in accordance with 
this code string synthesis signal, the code string synthe- 
sizer 74 connects the code strings from the decoders 62 
to 64 in a specific sequence to produce the same syntax 
as that in the case of no frame division, thus finally out- 
putting a synthetic or combined code string. 
[0005] In the foregoing Japanese Unexamined Patent 
Publication No. 7-298263, in accordance with the 
instruction from a division control unit, an image signal 
originally corresponding to one image is divided into 



division image signals corresponding to a plurality of 
areas which in turn, are separately encoded in a plural- 
ity of encoders according to MPEG (Moving Picture 
Experts Group), and the code strings outputted from the 
5 plurality of encoders are arranged to produce the origi- 
nal syntax. 

[0006] At this time, each of the encoders receives the 
size of the divided frame and the encoding condition to 
know the image signal, i.e., the area of the original 
10 image the encoder itself encodes, in other words, the 
image area in which the code string outputted by the 
encoder itself is finally displayed, before the MPEG 
encoding processing. 

[0007] However, in the case of displaying reduced 

15 moving images (each of which will be referred hereinaf- 
ter to as a thumbnail moving image) in arbitrary image 
areas of a multi-image raster (entire frame for multiple 
images), the displaying position is unknowable at the 
time that the MPEG encoding processing is conducted 

20 of each thumbnail moving image. 

[0008] For this reason, at the time of combining or 
integrating multiple images, an operation to designate a 
displaying position is needed for each of thumbnail mov- 
ing images encoded (which will be referred hereinafter 

25 to as an encoded thumbnail moving image). 

[0009] Meanwhile, for example, in the MPEG standard 
(for reference, a portion of a syntax of a video stream in 
MPEG is shown in Figs. 52A to 52D), a horizontal posi- 
tion at which a slice is displayed when decoded and 

30 reproduced is expressed by a variable-length code bit 
macroblock_address_increment (which will be referred 
hereinafter to as MB_Addr_lnc) of a leading macroblock 
in that slice. If making a change of the value of an 
MB_Addr_lnc section for the purpose of the disposition 

35 of the encoded thumbnail moving image at an arbitrary 
position, the number of bits of the MB_AddrJnc section 
forming a variable-length code varies. Consequently, (in 
the case of merely applying to MPEG as the processing 
to designate a displaying position for the encoded 

40 thumbnail moving image,) at the combination or integra- 
tion of moving images in a multi-image raster, a bit shift 
operation takes place by the bit number variation with 
respect to all the code strings following the 
MB_Addr_lnc section, which causes a large amount of 

45 calculation. 

[0010] Figs. 53A to 53C are illustrations available for 
describing this event. In the case that an encoded 
thumbnail moving image is disposed at a right and lower 
section of a multi-image raster comprising two divisions 

so (two columns) in the vertical direction and two divisions 
(two rows) in the horizontal direction as shown in Fig. 
53 A, the code string of the encoded thumbnail moving 
image A varies as shown in Fig. 53B. When viewing the 
entire moving image, the vertical position of a slice is 

55 specified with the last 1 byte in a slice_start_code sec- 
tion. 

[0011] On the other hand, the horizontal position of 
the slice is designated with the MB_Addr_lnc as men- 
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tioned above. Although the horizontal position of the 
slice varies from 1 to 5 due to the combination, if repre- 
senting as a variable-length code, MB_Addr_lnc = 1 sig- 
nifies T being 1 bit, and if expressing as a variable- 
length code MB_Addr_lnc = 5 signifies "0010" being 4 
bits, and hence, after the combination or composition, 
the bit string extends by 3 bit extra. Accordingly, as 
shown in Figs. 53B and 53C, the code strings succeed- 
ing MB_AddrJnc after the combination are located in a 
state of being shifted by the additionally increased 3 
bits. Usually, computers or storage units for common 
use in the computers gain access in units of bytes. For 
this reason, as shown in Fig. 53C, all the code strings 
succeeding MB_Addr_lnc of the leading macroblock 
require the bit operation. 

[0012] Furthermore, in the same MPEG standard, 
there is no guarantee for the code strings of the adja- 
cent macroblocks to be separated (byte-aligned) in units 
of bytes, while as a solitary instance, the byte alignment 
is accomplished by next_start_code 0 in Fi 9S- 52A to 
52D after the code string of the last macroblock of a 
group of macroblocks constituting a slice. When the 
macroblocks arranged in the aiming order for the loca- 
tion of an encoded thumbnail moving image at an arbi- 
trary position, the inter-macroblock connection crossing 
a byte border takes place, so that difficulty is experi- 
enced in decoding the code strings into a correct image. 
[0013] Still further, in terms of the code string in the 
last macroblock of a slice, the byte alignment is made 
with padding bits as mentioned above. In the case that 
another macroblock code string is connected succeed- 
ingly to the macroblock code string which contains 
added padding bits, a decoder decodes code strings on 
the assumption that there exists no padding bit in the 
code strings of the adjacent macroblocks except that 
the code strings form the last of the slice. Thus, the 
decoder can not achieve the decoding into a correct 
image by the padding bits in the code strings. 
[0014] Figs. 54A to 54C are illustrations available for 
explaining this situation. In this instance, encoded 
thumbnail moving images A and B shown in Fig. 54A 
are combined into a multi-image raster comprising one 
divisions (one row) in the vertical direction and two divi- 
sions (two columns) in the horizontal direction. In the 
case that this combination processing is realized in a 
manner that macroblocks organizing each of the 
encoded thumbnail moving images A and B are dis- 
posed at arbitrary positions within the multi-image 
raster, the disposition of the respective macroblocks 
within the multi-image raster is as shown in Fig. 54B t 
and as shown in the lowest portion of Fig. 54C, the 
arrangement of the code strings thereof is such that the 
code string corresponding to the slice header section of 
the slice M1 first appears, and then, is followed by the 
succeeding macroblock code strings in the order of the 
macroblocks A1, A2, B1 and B2. 
[0015] At this time, assuming that the last (end) mac- 
roblock A2 of the slice A1 is byte-aligned by padding 



bits, the code string of the slice M1 goes into a state in 
which a padding bit string is interposed in between the 
macroblocks A2 and B1. The result is that difficulty is 
encountered in correctly decoding an image. 

5 

SUMMARY OF THE INVENTION 

[0016] The present invention is for the purpose of 
eliminating the above-mentioned problems in a system 

10 in which thumbnail, moving images encoded according 
to various standards are displayed at arbitrary position 
on a multi-image raster, and it is therefore an object of 
the present invention to provide a moving-image com- 
posing system which is capable of, for example, in the 

15 MPEG standard, rewriting or changing a variable-length 
code bit portion representative of a displaying position 
of a slice to an arbitrary value at a high speed, combin- 
ing multiple images by only arranging macroblocks in a 
raster scanning sequence, simultaneously displaying 

20 thumbnail moving images with different sizes on one 
multi-image raster, and dividing the multi-image raster 
into encoded thumbnail moving images and then recon- 
structing a multi-image raster in an arbitrary sequence. 
[0017] Accordingly, in this invention, in consideration 

25 of the case that the location of reduced and encoded 
images before the combination within the composed or 
combined image are undecided, the encoding is done to 
facilitate the change of the positional information ther- 
eon, and if the change of the positional information 

30 thereof is needed at the combination, that change is 
facilitated, whereupon various combinations of images 
become feasible. 

[0018] Furthermore, in the case of the MPEG stand- 
ard, in the first place, in preparation for the variation in 

35 the number of bits of MB_AddrJnc resulting from the 
change of the horizontal coordinate of a slice, a code 
string (supplementary code string) is preliminarily out- 
putted to make up for the bit shortage so that a moving 
image combining means completes the rewriting 

40 processing of MB_AddrJnc (horizontal position infor- 
mation on a slice) comprising a variable-length code 
bit(s) for the multi-image combination in units of slices 
by performing simple calculations several times, thus 
enabling the speed-up of the production of a multi- 

45 image combined moving image. 

[0019] Still further, a moving image encoding means 
is provided which outputs 8 sets or pairs of 
extra_bit_slice and extra_information_slice as the afore- 
said supplementary code string, which allows a moving 

so image composing means to supply an arbitrary number 
of wanted or lacking bits from the supplementary code 
string and further to produce a macroblock_escape 
code string. This permits the rewriting of a variable- 
length code bit portion indicative of the displaying posi- 

55 tion of a slice to an arbitrary value. 

[0020] In the second place, the moving image com- 
posing means prevents the head and end of a macrob- 
lock code string from crossing a byte border and 
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previously produces a macroblock code string undergo- 
ing such processing as to make a correct code string as 
an MPEG video sequence, whereupon the moving 
image composing means can produce a code string of 
a desired multi-image combined moving image by only 5 
connecting macroblock code strings in the raster scan- 
ning order, which contributes to the speed-up of the 
multi-image combined moving image production. 
[0021] In the third place, the moving image composing 
means is capable of combining encoded thumbnail 
moving images with different sizes in units of slices, and 
consequently, the moving image composing system can 
offer a code string of a multi-image combined moving 
image in which encoded thumbnail moving images with 
different sizes mixedly exist. 

[0022] In the fourth place, an MPEG code string 
already constructed as a multi-image combined moving 
image is separated into a plurality of original encoded 
thumbnail moving images, and the encoded thumbnail 
moving images separated are again disposed at arbi- 
trary positions, so that a list of combinations and orders 
of programs desired by viewers can be reconstructed 
from multi-image combined code strings such as a list of 
multi-image programs predetermined in a satellite 
broadcasting or the like. 

[0023] In the fifth place, an image composing means 
rewrites an information value on an encoded image to 
output a multi-image combined image stream in which a 
code string of each of encoded images is a correct infor- 
mation value as an image encoding format and a code 
string sequence is a correct sequence as the image 
encoding format. Whereupon, even when the code 
length variation of information occurs due to the change 
of the information value defined with a variable-length 
code, the change of the information value is made pos- 
sible without requiring a extremely large amount of 
processing such as a bit shift operation, which achieves 
the speed-up of the multi-image combined image 
stream. 

[0024] Furthermore, at this time, since the code string 
sequence of the multi-image combined image produced 
by the image composing means is correct as the MPEG 
format, the decoding, the reproduction and the display- 
ing become feasible through the use of an image decod- 
ing means for decoding an MPEG code string, or an 
image decoding unit (generally called MPEG decoder). 
[0025] In the sixth place, the moving image compos- 
ing system has a construction in which a delay means 
does not exist in a process, and the image composing 
means receives at least one encoded image produced 
without a single passage through the delay means. 
Whereupon, a multi -image combined image such as a 
combination of a real-time live picture such as an image 
coming as an inputted image from a camera and an 
accumulated image can be displayed, and the delay 
reproduction of a live picture portion by the delay means 
is possible while the reproduction and display of a multi- 
image combined image undergoing the adjustment of 



the delay time due to the frame thinning processing are 
feasible. 

[0026] In the seventh place, a layout table changing 
means is provided to change the value of multi-image 
layout information within a layout table at an arbitrary 
timing, and the image composing means refers to the 
value of the multi-image layout information after the 
change, thereby allowing the dynamic change of the 
image layout of a multi-image combined image stream 
to be outputted. Thus, it is possible to dynamically 
change the disposition of areas of the multi-image com- 
bined image, the size of each of the areas, the encoded 
image to be displayed in each of the areas, and others. 
[0027] In the eighth place, an image composing 
means is provided which produces a multi-image com- 
bined image stream when receiving at least one 
encoded image produced by the encoding processing 
based upon an image encoding format, and when there 
is a need to change an information value defined as a 
variable-length code in the image encoding format, the 
image composing means uses an extended code string 
portion defined as a code string not affecting the images 
in the image encoding format to output the extended 
code string into the multi-image combined image 
stream, thereby changing that information value in the 
encoded image to a different value. Whereupon, irre- 
spective of the variation of the information code length 
occurring due to the change of the information value 
defined as a variable-length code, the change of the 
information value becomes feasible without conducting 
an extremely large amount of processing such as the bit 
shift operation, which contributes to the speed-up of the 
multi-image combined image producing processing. 
[0028] In addition, since there is no need for the image 
encoding means which produces an encoded image to 
be inputted to this image composing means to conduct 
particular processing for realizing the aforesaid speed- 
up, it is possible to employ a common image encoding 
means which performs the encoding processing on the 
basis of the image encoding format. Besides, it is possi- 
ble to accomplish the speed-up of the combining 
processing among common encoded images according 
to the image encoding format 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0029] The object and features of the present inven- 
tion will become more readily apparent from the follow- 
ing detailed description of the preferred embodiments 
taken in conjunction with the accompanying drawings in 
which: 

Fig. 1 is a schematic illustration of an arrangement 
of a moving image composing system according to 
the present invention; 

Fig. 2 is a block diagram showing a moving image 
composing system according to a first embodiment 
of this invention; 
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Fig. 3 is an operational flow chart showing an oper- 
ation of a moving image encoding means according 
to the first embodiment of this invention; 
Fig. 4 is an illustration of an example of block divi- 
sion of a thumbnail moving image in the first s 
embodiment of this invention; 
Fig. 5 is an illustration of an example of a code 
string to be outputted from the moving image 
encoding means in the first embodiment of this 
invention; 10 
Fig. 6 is an illustration of an example of a list of 
encoded thumbnail moving images accumulated in 
an accumulating means in the first embodiment of 
this invention; 

Fig. 7 is an illustration of an example of a multi- is 
image combined moving image produced in answer 
to a request a moving image listing means issues to 
the moving image composing means in the first 
embodiment of this i nvention ; 

Fig. 8 is an operational flow chart showing an oper- 20 
ation of the moving image composing means 
according to the first embodiment of this invention; 
Fig. 9 is an illustration of an example of loop 
processing in the interior of the moving image com- 
posing means according to the first embodiment of 25 
this invention; 

Figs. 10A and 10B are illustrations of slice code 
string readout processing from the accumulating 
means by the moving image composing means 
according to the first embodiment of this invention; 30 
Fig. 1 1 consisting of Figs. 1 1 A to 11 C is an illustra- 
tion of an example of slice vertical position informa- 
tion rewriting processing in the first embodiment of 
this invention (vertical position = 3); 
Fig. 12 consisting of Figs. 12A to 12D is an illustra- 35 
tion of an example of slice horizontal position infor- 
mation rewriting processing in the first embodiment 
of this invention (horizontal position = 4); 
Fig. 13 consisting of Figs. 13Ato 13D is an illustra- 
tion of an example of slice horizontal position infor- 40 
mation rewriting processing in the first embodiment 
of this invention (horizontal position = 7); 
Fig. 14 consisting of Figs. 14Ato 14D is an illustra- 
tion of an example of slice horizontal position infor- 
mation rewriting processing in the first embodiment 45 
of this invention (horizontal position = 13); 
Fig. 15 consisting of Figs. 15A to 15D is an illustra- 
tion of an example of slice horizontal position infor- 
mation rewriting processing in the first embodiment 
of this invention (horizontal position = 37); so 
Fig. 1 6 is a block diagram showing a moving image 
composing system according to a second embodi- 
ment of this invention; 

Fig. 17 is an operational flow chart showing an 
operation of a moving image encoding means in the 55 
second embodiment of this invention; 
Fig. 18 consisting of Figs. 18A to 18C is an illustra- 
tion of an example of a slice header code string in 



the second embodiment of this invention; 
Fig. 19 consisting of Figs. 19A to 19C is an illustra- 
tion of an example of an operation of a work buffer 
in the second embodiment of this invention; 
Fig. 20 consisting of Figs. 20A to 20C is an illustra- 
tion of an example of an operation of the work buffer 
at the MPEG encoding processing by the moving 
image encoding means in the second embodiment 
of this invention; 

Fig. 21 is an illustration of the relationship between 
the number of insertion of macroblock_stuffing and 
the values of a variable b in the second embodi- 
ment of this invention; 

Fig. 22 is an operational flow chart showing an 
operation of a moving image composing means 
according to the second embodiment of this inven- 
tion; 

Fig. 23 is an illustration of an example of loop 
processing in the interior of the moving image com- 
posing means according to the second embodi- 
ment of this invention; 

Fig. 24 is an illustration of an example of an image 
composing processing sequence corresponding to 
one multi-image raster in the second embodiment 
of this invention; 

Fig. 25 is a block diagram showing a moving image 
composing system according to a third embodiment 
of this invention; 

Fig. 26 is an operational flow chart showing an 
operation of a moving image encoding means in the 
third embodiment of this invention; 
Fig. 27 is an illustration of an example of a multi- 
image arrangement in the third embodiment of this 
invention; 

Fig. 28 is an illustration of an example of initializa- 
tion of a two-dimensional array variable V in the 
third embodiment of this invention; 
Fig. 29 is an illustration of an example of loop 
processing in the interior of the moving image com- 
posing means according to the second embodi- 
ment of this invention; 

Fig. 30 is an illustration of a skip slice code string in 
the third embodiment of this invention; 
Fig. 31 is a block diagram showing a moving image 
composing system according to a fourth embodi- 
ment of this invention; 

Fig. 32 is an operational flow chart showing an 
operation of a multi-image combined moving image 
dividing means in the fourth embodiment of this 
invention; 

Fig. 33 is an illustration of an example of a format of 
an image layout table in the fourth embodiment of 
this invention; 

Fig. 34 is an illustration of an example of a multi- 
image combined moving image produced in 
response to a request a moving image listing 
means issues to the moving image composing 
means in the fourth embodiment of this invention; 
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Fig. 35 is an illustration of an example of the 
progress of an operation of the multi-image com- 
bined moving image dividing means in the fourth 
embodiment of this invention; 

Fig. 36 consisting of Figs. 36A to 36D is an illustra- s 
tion of an example of slice code string conversion 
processing by a moving image re-encoding means 
in the fourth embodiment of this invention; 
Fig. 37 is a block diagram showing an construction 
of a moving image composing system according to w 
a fifth embodiment of this invention; 
Fig. 38 is an illustration of an example of a multi- 
image combined image in the fifth embodiment of 
this invention; 

Fig. 39 consisting of Figs. 39A and 39B-1 to 39B-3 is 
is an illustration of a code string sequence of a 
multi-image combined image in the fifth embodi- 
ment of this invention; 

Fig. 40 is a block diagram showing a construction of 
a moving image composing system according to a 20 
sixth embodiment of this invention; 
Fig. 41 is an illustration of input paths of encoded 
images to an image composing means in the sixth 
embodiment of this invention; 

Fig. 42 is an illustration of a reproduced multi-image 25 
combined image in the sixth embodiment of this 
invention; 

Fig. 43 is a block diagram showing a construction of 
a moving image composing system according to a 
seventh embodiment of this invention; 30 
Rg. 44 is an illustration of image structures of a 
multi-image combined image before and after the 
change of the contents of a layout table in the sev- 
enth embodiment of this invention; 
Fig. 45 is an illustration of combining processing 35 
among encoded images in an image composing 
means in the seventh embodiment of this invention; 
Rg. 46 is a block diagram showing a construction of 
a moving image composing system according to an 
eighth embodiment of this invention; 40 
Fig. 47 is an illustration of an example of a code 
string to be outputted from an image encoding 
means in the eighth embodiment of this invention; 
Rg. 48 is an illustration of a list of encoded images 
accumulated in an accumulating means in the 45 
eighth embodiment of this invention; 
Rgs. 49A and 49B are illustrations of an example of 
slice code string readout processing from the accu- 
mulating means by an image composing means in 
the eighth embodiment of this invention; so 
Rg. 50 consisting of Rgs. 50A to SOD is an illustra- 
tion of an example of rewriting processing of slice 
horizontal position information in the eighth embod- 
iment of this invention; 

Rg. 51 is a block diagram showing an arrangement ss 
for obtaining a code string based on moving image 
division, identical in syntax with a code string pro- 
duced with no moving image division in the related 



art; 

Rgs. 52A to 52D are illustrations of a portion of a 
syntax of a video stream in MPEG; 
Rgs. 53A to 53C are illustrations of an event occur- 
ring at the rewriting of slice horizontal position infor- 
mation; and 

Rgs. 54A to 54C are illustrations of an event taking 
place at the connection between macroblocks not 
undergoing byte alignment. 

DETAILED DESCRIPTION OF PREFERRED EMBOD- 
IMENTS 

[0030] Embodiments of the present invention will be 
described hereinbelow with reference to the drawings. 
This invention is not limited to these embodiments, but it 
is intended to cover a variety of embodiments, which do 
not constitute departures from the spirit and scope of 
the invention. 

[0031] Prior to describing the embodiment of this 
invention, a brief description will be made hereinbelow 
of an arrangement of a moving image composing sys- 
tem according to this invention taken as a whole. Rg. 1 
is an illustration of a general construction of the moving 
image composing system of this invention. In Fig. 1 , 
moving image data is inputted to a moving image 
receiving means 101, and then put in a moving image 
reducing means 102 for reducing a moving image to a 
given size, and further put in a moving image encoding 
means 103 for encoding a thumbnail moving image into 
an MPEG code to produce an encoded thumbnail mov- 
ing image, with the encoded thumbnail moving images 
being stored and accumulated in an accumulating 
means 105. A moving image composing means 106 is 
for combining or integrating a plurality of encoded 
thumbnail moving image to produce a multi-image com- 
bined moving image code string, and is in connection 
with a combination work memory 104 and a moving 
image listing means 107. The combination work mem- 
ory 104 is for temporarily storing a slice code string at 
the time of rewriting positional information, while the 
moving image listing means 107 is for drawing up a list 
of a combination and order of moving images constitut- 
ing a multi -image raster, that is, for making a request to 
the moving image composing means 106 for the com- 
position of a given multi-image combined moving 
image. The multi-image combined moving image com- 
posed by the moving image composing means 106 is 
decided in a moving image decoding means 108 and 
subsequently displayed on a moving image displaying 
means 109. 

(First Embodiment) 

[0032] Fig. 2 is a block diagram showing a construc- 
tional feature of a moving image composing system 
according to a first embodiment of this invention. 
[0033] As shown in Fig. 2, the moving image compos- 
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ing system according to the iirst embodiment is made 
up of a moving image composing server 201 for manag- 
ing a plurality of encoded thumbnail moving images 
accumulated and for conducting the multi-image com- 
posing processing, and a reproduction terminal 208 for 5 
reproducing and displaying the multi-image combined 
moving image from the moving image composing 
server 201 . The moving image composing server 201 
comprises a moving image receiving means 202 for 
receiving moving images to be combined into a multi- ro 
image combined moving image, a moving image reduc- 
ing means 203 for reducing the received moving images 
to a given or predetermined size to produce thumbnail 
moving images, a moving image encoding means for 
MPEG-encoding the thumbnail moving images to ere- 75 
ate encoded thumbnail moving images, a composition 
work memory 205 for temporarily storing slice code 
strings at rewriting of positional information on the 
encoded thumbnail moving images, an accumulating 
means 206 for storing and accumulating the encoded 20 
thumbnail moving images, and a moving image com- 
posing means 207 for combining a plurality of encoded 
thumbnail moving images to produce a multi-image 
combined moving image code string. The reproduction 
terminal 208 comprises a moving image listing means 25 

209 for drawing up a list or table of a combination and 
order of moving images organizing a multi-image com- 
bined moving image, a moving image decoding means 

210 for decoding a composed multi-image combined 
moving image, and a moving image displaying means 30 

21 1 for displaying the decoded moving image. 
[0034] For the moving image composing server 201 
and the reproduction terminal 208, a common operating 
system operates, which is, for example, a computer 
such as a personal computer and a work station, while 35 
each of the moving image receiving means 202, the 
moving image reducing means 203, the moving image 
encoding means 204, the moving image composing 
means 207, the moving image listing means 209, the 
moving image decoding means 210 and the moving 40 
image displaying means 211 is constructed as a pro- 
gram or dedicated hardware operable under the control 
of a general-purpose or dedicated operating system 
within the computer. 

[0035] Secondly, a description will be given hereinbe- 45 
low of a method of encoding moving images and a 
method of combining a plurality of moving images. 
[0036] The description will begins with the moving 
image data encoding processing. 

[0037] Moving image data to be combined into a multi- so 
image combined moving image is previously taken 
through the moving image receiving means 202 into the 
interior of the moving image composing server 201 , and 
then, is reduced to a predetermined size by means of 
the moving image reducing means 203 to be converted 55 
into a thumbnail moving image. Subsequently, this 
thumbnail moving image is MPEG-encoded in the mov- 
ing image encoding means 204 in the form allowing the 
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multi-image combination, and is accumulated in the 
accumulating means 206 as an encoded thumbnail 
moving image to be used for the multi-image combina- 
tion. 

[0038] Fig. 3 illustrates an operational flow taken until 
a thumbnail moving image is converted into an encoded 
thumbnail moving image based on MPEG for the multi- 
image combination. 

[0039] In Fig. 3, the operational flow proceeds to a 
step 301 (S301 in the illustration) to read out image data 
corresponding to one thumbnail moving image frame, 
then followed by a step 302 to divide the one-frame 
image data along a horizontal direction into N slices, 
and further, followed by a step 303 to output a slice 
header code string of the slice n as a portion of an 
encoded thumbnail moving image code string. Follow- 
' ing this, a step 304 is executed to divide the slice n into 
M macroblocks, then followed by a step 305 to output 
the MB_Addr_lnc of the macroblock m as a portion of 
the encoded thumbnail moving image code string, and 
further, followed by a step 306 to MPEG-encode (DCT, 
quantization, variable-length encoding) the macroblock 
m to output the resultant code string as a portion of the 
encoded thumbnail moving image code string. 
[0040] After this, the operational flow advances to a 
step 307 to check whether or not the encoding of all the 
macroblocks in the slice n reaches completion. If reach; 
ing completion, a step 308 follows to conduct the 
processing of next_startt-code () in Fig. 521 D, and sub- 
sequently, a step 309 follows to check whether or not 
the encoding of all the N slices reaches completion. If 
reaching completion, a step 310 follows to check 
whether or not to process all the frames of the thumbnail 
moving image. In this case, the variable m is used for 
identifying the macroblock processed, while the variable 
n is for identifying the slice. 

[0041] Fig. 4 is an illustration of an example of division 
of each of frame images of a thumbnail moving image 
into small blocks in the case that the moving image 
encoding means 204 encodes the thumbnail moving 
image as encoded thumbnail moving image (steps 302 
and 304 in Fig. 3). 

[0042] Fig. 5 is an illustration of a code string output- 
ted to permit the multi-image combination in units of 
slices when the moving image encoding means 204 
MPEG-encodes each of the slices of a thumbnail mov- 
ing image in the steps 303, 305, 306 and 308 in Fig. 3. 
Incidentally, the instance shown in Fig. 5 is based upon 
the division examples into the slices and macroblocks in 
Fig. 4. 

[0043] As seen in (a) and (e) of Fig. 5. in the case of 
outputting a slice code string of the slice n in the step 
303, slice_start_code is outputted in a state where the 
last one byte is set as the value M n M . 
[0044] Furthermore, as shown in (b) of Fig. 5, when 
multi-image combination is made in units of slices, in 
preparation for the event that the number of bits of 
MB_Addr_lnc varies due to the change of horizontal 
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coordinate of the slice, a supplementary code string for 
/ supplying the wanted bits is outputted in advance as 
/ extra_bit_slice and extra_information_slice. This sup- 
/ plementary code string utilizes a portion prescribed 
/ even in the MPEG standard in Figs. 52A to 52D, and the 
/ decoder is capable of normally decoding an encoded 
/ thumbnail moving image with such a code string as an 
MPEG video sequence. 

[0045] In the example of Fig. 5. as the supplementary 
code string, 8 sets of extra_bit_slice and 
extra_Jnformation_slice, that is, a code string compris- 
— ^ing 72 bits in total, are outputted in a state where all the 
bits are set as "1" (the reason for the 8 sets of 
extra_bit_slice and extra_information_slice will be 
described herein later). 

[0046] Still further, as shown in (c) of Fig. 5, in the step 
305, the value "1" is designated to MB_AddrJnc of the 
leading macroblock of the slice, and a variable-length 
code string "1 " of 1 -bit length representative of value "1 
is outputted. Each of code string shown in (d) of Fig. 5 
are a code string obtained by conducting the common 
MPEG encoding processing (DCT, quantization, move- 
ment estimation, variable-length encoding and byte 
alignment processing) in steps 306 and 308. 
[0047] Incidentally, when the multi-image combination 
is made in units of slices, a portion required as an 
encoded thumbnail moving image is a code string por- 
tion following the slice layer in Fig. 5. However, in a man- 
ner that code strings (sequence header, GOP header, 
picture header and others) needed as the MPEG video 
sequence are outputted in addition to the code strings 
of the aforesaid slice section, an encoded thumbnail 
moving image composed of these code strings can be 
decoded as a single MPEG video sequence and repro- 
duced to be displayed. 

[0048] In the above description made with reference 
to (a), (c) and (e) of Fig. 5, although slice_start_code 
and MB_AddrJnc are set to the values whereby an 
encoded thumbnail moving image is decoded as a sin- 
gle MPEG video sequence and reproduced to be dis- 
played, in fact, any value is acceptable at the encoding 
operation. 

[0049] Moreover, as described with reference to (b) of 
Fig. 5, although, in preparation for the case that the 
number of bits of MB_Addr_lnc varies due to the varia- 
tion of the slice horizontal coordinate, a code string for 
supplying wanted bits is outputted in advance as 
extra_bit_slice and extra_information_slice, Jt is also 
appropriate to previously output a code string such as 
macroblock_stuffing which is available for accomplish- 
ing the supplementation processing in connection with 
the bit number variation in the MB Addr Inc section . 
[0050] Through the above-described process, the 
moving encoding means 204 outputs an encoded 
thumbnail moving image code string for the multi-image 
combination in units of slices. 

[0051] A description will be taken hereinbelow of the 
processing to combine a plurality of encoded thumbnail 



moving images into a multi-image combined moving 
image. 

[0052] Fig. 6 is an illustration of one example of each 
of encoded thumbnail moving images accumulated in 

5 the accumulating means 206. As shown in Fig. 6, five 
encoded thumbnail moving images A, B, C, D and E 
accumulate in the accumulating means 206. At this 
time, let it be assumed that each of the five encoded 
thumbnail moving images A, B, C, D and E is divided 

10 horizontally into two slices each of which in turn, com- 
prises three macroblocks. 

[0053] The moving image listing means 209 draws up 
a list of a combination and order of the encoded thumb- 
nail moving images accumulated in the accumulating 

is means 206. A one example, the moving image listing 
means 209 produces a list depending upon the hit rate 
ranking , of retrieval keys. A description here will be 
made assuming that the moving image listing means 
209 issues a request to the moving image composing 

20 means 207 for the composition of a multi-image com- 
bined moving image comprising two divisions (two col- 
umns) in the vertical direction and two divisions (two 
rows) in the horizontal direction as shown in Fig. 7. 
[0054] Fig. 8 shows an operational flow to be taken 

25 until the moving image composing means 207 com- 
bines a plurality of encoded thumbnail moving images 
into a multi-image combined moving image and outputs 
an MPEG code string. 

[0055] The operational flow starts at a step 801 to per- 

30 form the initialization for the multi-image combination, 
and then, advances to a step 802 to output an MPEG 
video sequence header code string so that a multi- 
image combined moving image can be decoded as an 
MPEG video sequence, and subsequently, proceeds to 

35 a step 803 to output GOP header and picture header 
code strings so that it can likewise be decoded as the 
.MPEG video sequence. Further, a step 804 follows to, 
read out a slice code,strina.from-each-ot-the-encoded 
thumbnail moving images, then fo llow.ed.bv a stftn.805- 

40 to rewrite positional information for disp layingJhejieaci. 
slice at a de sired Place within the_multi-image_r_aster. 
and further followed by, a r step^806^to,check.the T countec 
yal u e_ foJL Scan ni ng _th L^^encj^ecL^thumbnail^moving 
image in the horizontal jdirectlo nsu. Still further, a step 

45 807 is implemented to check the slice counter value to 
be used in each of steps 904 and 905, and a step 808 is 
executed to check the counter value for scanning the 
encoded thumbnail moving image in the vertical direc- 
tions, and further, a step 809 is done to check whether 

so the multi-image combination is terminated or not. 

[0056] In response to a request for the rnulti -image 
combination shown in Fig. 7, the moving image com- 
posing means 207 first undergoes the following initiali- 
zation processing. That is. because of the multi-image 

55 raster comprising two divisions in the vertical direction 
and two divisions in the horizontal direction, the variable 
X representative of the number of images in the hori- 
zontal direction is set to 2, while the variable Y indicative 
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of the number of images in the vertical direction is set at 
2. In addition, the variable N representative of the 
number of slices of each of encoded thumbnail moving 
images to be displayed on each of image areas within 
the mult-image raster is initialized to 2, while the varia- 
ble M indicative of the number of macroblocks constitut- 
ing each of the slices is initialized to 3. Moreover, the 
two-dimensional array variable V (the first subscript rep- 
resents a horizontal image area of the aforesaid image 
areas, and the second subscript signifies a vertical 
image area thereof) indicating which encoded thumb- 
nail moving image is located at one position within the 
multi-image raster is initialized as follows. 

V[1 ,1] = B, V[2, 1] = D, V[1 , 2] = E, V[2, 2] = A 

[0057] Fig. 9 is an illustration of expressing loop oper- 
ations in loops 810,811 and 812 (L810, L811, L812 in 
the illustration) in a C language, where the variable x 
takes the horizontal position of the image, being proc- 
essed, as a loop counter value, and the variable y takes 
the vertical position thereof as a loop counter value, 
while the variable n assumes the slice number of an 
encoded thumbnail moving image under processing as 
a loop counter value. In the innermost loop of these 
three loops, the processing of the steps 804 and 805 
are put into practice. 

[0058] Figs. 10A and 10B show the slice code string 
readout processing the moving image composing 
means 207 performs in the step 804 in Fig. 8. In the step 
804, a slice portion code string, designated by the vari- 
able n, of a frame specified by the variable f is read out 
from the encoded thumbnail moving image specified by 
a variable V[x, y] [(a) of Fig. 12]. For instance, when x - 
2, y = 2 and n = 1 , the code string of the slice 1 A is read 
out in the step 804. the slice string read out is stored in 
the composition work memory in which the leading 
address of the buffer is pointed to by a variable Buf [(b) 
of Fig. 12]. 

[0059] In the case of the MPEG standard, the code 
string of the slice layer is prescribed such that the code 
string starts from a byte border and the end of the code 
string terminates in a state of being byte-aligned, and 
the moving image encoding means 204 outputs such a 
slice code string. For this reason, the slice code string is 
wholly read from its head to its end in units of bytes and 
is stored in the composition work memory. 
[0060] The rewriting processing of the slice displaying 
position information to be done by the moving image 
composing means 207 in the step 805 is as follows. 
[0061] The moving image composing means 207 cal- 
culates, in accordance with the following equation, the 
, coordinate in the multi-image raster on which the slice 
code string within the composition work memory is dis- 
played. 

Horizontal Position: (x - 1) * M + 1 
Vertical Position: (y - 1 ) * N + n 
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[0062] In the case of the aforesaid slice 1 A, since x = 
2, y = 2 and n = 1, the result is that the horizontal posi- 
tion = 4 and the vertical position = 3. 
[0063] Fig. 11 is an illustration of the rewriting 

s processing of the vertical position information on the 
slice 1 A within the composition work memory. In Fig. 1 1 , 
the section indicated by Fig. 1 1 A shows a state the code 
string of the slice 1A is stored in the composition work 
memory in the step 804, while the section of Fig. 11B 

io illustrates a detailed indication of a slice_start_code 
portion including a code string of the vertical position 
information on a slice in units of bits. Since the slice ver- 
tical position information is represented by the fourth 
byte of slice_start_jcode that is, buf[3], in the step 805, 

is the substitution processing is conducted as follows in 
order to set the vertical position at 3. 

buf[3] = 3 

20 [0064] In Fig. 11, the section indicated by Fig. 11C 
denotes the code string of the slice_start_code portion 
after the rewriting of the vertical position information by 
the substitution processing. 

[0065] Likewise, Fig. 12 shows one example of the 

25 rewriting processing of the horizontal position Informa- 
tion on the slice 1 A within the composition work mem- 
ory. In Fig. 12, the section of Fig. 12A illustrates a state 
where the code string of the slice 1A is read into the 
composition work memory in the step 804, while the 

30 section of Fig. 12B illustrates a detailed indication in 
units *of bits. Although, a new horizontal position 
(MB__Addr_lnc) of the slice 1 A assumes 4, if expressing 
in a variable-length code, the horizontal position 4 is 
"001 r forming four-bit length. The moving image 

35 encoding means 204, which produces encoded thumb- 
nail moving; images, outputs MB_Addr_lnc as "1", that 
is, one-bit length as shown in Fig. 12B of Fig. 12. Thus, 
the moving image composing means 207 makes up for 
the wanted three bits from the supplementary code 

40 string portion. First, as shown in Fig. 12C of Fig. 12, the 
bytes identical in number to the wanted bits are thin- 
ning-processed (In the instance of the slice 1A, the 
number of lacking bits is 3, and therefore, three bytes 
are taken in a thinning way). 

45 [0066] Secondly, after the code string corresponding 
to 1 0 bytes from buf[0] to buf[9] not undergoing the thin- 
ning processing is outputted, new horizontal position 
information rewriting processing is conducted as shown 
in the sections Figs. 12C and 12D of Fig. 12. 

so 

buf[13] = buf[13]&0x01 + 0xc6 

[0067] In this equation, buf[13]&0x01 is for a filter cal- 
culation not to rewrite the lower 1 bit of buf[13]. 
55 [0068] After the rewriting of the horizontal position, all 
the slice code strings succeeding buf [13] are outputted 
so as not to output the code string of the byte portion 
taken out by the thinning. Following the output of all the 
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slice code strings, the composition work memory is set 
into a vacant condition. 

[0069] Through the above-described process, the 
moving image composing means 207 completes the 
rewriting processin g of the va riable:leogtb-code,bitpor- 
tion indicative ojthe slice horizon tal.positiQnJnformatjon 
by only making simple calc ulation se veral times , and 
outputs a multi-image combined moving image code 
string produced by combining multiple images in units of 
slice code strings. The multi-image combined moving 
image code string outputted from the moving image 
composing means 207 is decoded as a moving image in 
the moving image decoding means 210, and that pic- 
ture is displayed on the moving image displaying means 
211. 

[0070] The reason for the use of 8 sets of 
extra_bit_slice and extra_irrformation_slice will be men- 
tioned hereinbelow, 

[0071] The extra_bit_slice is 1-bit length while the 
extra_information_slice is 8-bit length, and when these 
are paired, the code length reaches 9 bits. When 1 byte 
(- 8-bit length) is taken out from the supplementary 
code string portion of 9-bit length by means of the thin- 
ning, 1 bit remains there. This 1 bit can be allocated for 
the missing-bit supplementation at the variation in the 
number of bits of the MB_Addr_lnc portion. 
[0072] In terms of the aforesaid sets, if 1 byte per 1 set 
is taken out by the thinning, 1 bit is available for the sup- 
plementation, and therefore, when 7 bytes taken out 
from 8 sets, 8 bits are available for the supplementation. 
In the case of needing 9 or more supplementry bits, the 
number of bytes to be taken out by the thinning may be 
decreased on all such occasions. For instance, if sup- 
plying 14 bits, only 5 bytes are first taken out, so that 5 
bits are compensable, and then, 9 bits corresponding to 
1 set are further allocated as the supplementary bits, so 
that 14 bits in total are obtainable as the supplementary 
bits. 

[0073] Accordingly, since the combinations of 
extra_bit_sliceand extra_information_sliceare made by 
8 sets, it is possible to supply an arbitrary number of 
wanted bits, and since 8 sets form a minimum value, the 
supplementary code string can suppress the increase in 
the bit rate of the moving image to a minimum. 
[0074] As described above, according to this embodi- 
ment, there is provided a moving image composing sys- 
tem comprising the moving image receiving means for 
receiving moving images to be combined into a multi- 
image combined moving image, the moving image 
reducing means for reducing the received moving 
images to a predetermined size to produce thumbnail 
moving images, the moving image encoding means for 
MPEG-encoding the thumbnail moving images to pro- 
duce encoded thumbnail moving images, the accumu- 
lating means for storing and accumulating the encoded 
thumbnail moving images encoded according to MPEG, 
the moving image composing means for combining a 
plurality of encoded thumbnail moving images accumu- 



lated in the accumulating means to produce a multi- 
image combined moving image code string, the compo- 
sition work memory to be used for when the moving 
image composing means temporarily stores slice code 

5 strings, the moving image listing means for drawing up 
a list of a combination and order of moving images con- 
stituting a multi-image combined moving image, the 
moving image decoding means for decoding the multi- 
image combined moving image code string produced by 

10 the composition, and the moving image displaying 
means for displaying the decoded moving image, 
wherein, a variation in the number of bits of an 
MB_Addr_lnc occurs due to a change of the horizontal 
coordinate of a slice, the moving image encoding 

75 means previously outputs a supplementary code string 
to be used for supplying wanted bits, and the moving 
image composing means supplies the wanted bits from 
the supplementary code string and rewrites the value of 
MB_Addr_lnc, thus producing a multi -image combined 

20 moving image code string. 

[0075] This can omit the bit shift operation for all the 
succeeding code strings due to the variation in the 
number of bits at the composition of the multi-image 
combined moving image, and more specifically, the 

25 rewriting processing of the variable-length bit portion 
representing the slice horizontal position information 
can be completed by only performing simple calculation 
several times, and consequently, the multi-image com- 
bined moving image is producible at a high speed, 

30 which provides a great practical effect. 

[0076] More concretely, owing to the moving image 
encoding means having a function to output 8 sets 
extra_bit_slice and extra_information_slice as a supple- 
mentary code string, the moving image composing 

35 means can supply an arbitrary number of wanted bits 
from the aforesaid supplementary code string and can 
produce a macroblock_escape code string, so that the 
rewriting of the variable-length code bit portion repre- 
senting the slice displaying position to an arbitrary value 

40 becomes feasible, which offers a great practical effect. 
[0077] Incidentally, although this embodiment has 
been described in the case that the horizontal positions 
of the slice disposition are four in number, the rewriting 
processing is also possible even in the case of taking a 

45 different slice disposition. Figs. 13, 14 and 15 show the 
MB_Addr_lnc rewriting processing in the case that the 
horizontal positions are 7,13 and 37 in number, respec- 
tively. Although a 11 -bit code string called 
macrob!ock_escape should be outputted in the case 

so that the disposing positions are 34 in number, even in 
this case, in a manner of making the numerical change 
of the number of bytes to be taken out by thinning, the 
filter calculation and others, the same effects are attain- 
able. 

55 [0078] Furthermore, although, in the above descrip- 
tion of this invention, the variable-length code bit 
MB_AddMnc is used as a portion representative of the 
horizontal position of a slice code string and a series of 
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processing of supplying wanted bits resulting from the 
rewriting of the MB_Addr_lnc by simple byte thinning of 
a supplementary code string outputted in advance to 
the slice header portion is done taking the MPEG video 
sequence into consideration, this invention is also appli- 
cable to a standard other than the MPEG standard. 
[0079] That is, in a standard that positional information 
indicative of a position at which one code string is dis- 
played within the image is given with a variable-length 
code bit(s) when that code string is decoded into an 
image, if an area which can output an extended code 
string not affecting the image at decoding is prescribed 
as a standard in front or rear of the code string portion 
representative of the positional information, or in both, 
that area is used as a supplementary code string, so 
that the same effects are obtainable even in a standard 
other than the MPEG standard. 

(Second Embodiment) 

[0080] A description will be made hereinbelow of a 
second embodiment of this invention. In the explanation 
of the second embodiment, the same numerals are 
assigned to the same parts or sections as those 
described in the first embodiment, and the description 
thereof will be omitted for brevity. 
[0081 ] Fig. 1 6 is a block diagram showing an arrange- 
ment of a moving image composing system according 
to the second embodiment of this invention. In Fig. 16, 
numeral 1601 represents a moving image encoding 
means for MPEG-encoding thumbnail moving images to 
produce encoded thumbnail moving images, numeral 
1602 designates an encoding work memory for tempo- 
rarily storing the code strings produced by the MPEG 
encoding operation, and numeral 1603 depicts a mov- 
ing image composing means for combining a plurality of 
encoded thumbnail moving images to produce a multi- 
image combined moving image code string. 
[0082] A description will be given hereinbelow of 
methods for the moving image encoding and for the 
combination of a plurality of moving images to be taken 
in this moving image composing system. 
[0083] The description begins at the moving image 
encoding processing. 

[0084] Fig. 17 shows an operational flow to be taken 
until the moving image encoding means 1601 converts 
the thumbnail moving image into an encoded thumbnail 
moving image based on MPEG for a multi-image combi- 
nation. 

[0085] In Fig. 17, in a step 1701, a slice header code 
string of a slice n is outputted as a portion of an 
encoded thumbnail moving image code string, and in a 
step 1 702, the contents of the encoding work memory is 
marip pmnty the nrnrPRsinn for thp initialization of 
the encoding work memory. Further, in a step 1703, a 
macroblock m is MPEG-encoded (DCT, quantization, 
variable-length encoding), and a code string produced 
by the encoding is outputted to the encoding work mem- 



20 

ory, and in a step 1704, the code string within the 
encoding work memory is byte-aligned, and in a step 
1705, the code string within the encoding work memory 
is outputted as a portion of an encoded thumbnail mov- 

5 ing image code string. 

[0086] Fig. 1 8 is an illustration of an example of a slice 
header code string in the step 1702. In Fig. 18, the sec- 
tions indicated by Figs. 18A and 18B illustrate an exam- 
ple of a code string outputted by a common MPEG 

10 encoding means. As seen from Figs. 18A and 18B of 
Fig. 18, the head of a macroblock 1 crosses the byte 
border. Thus, in this embodiment, the moving image 
encoding means 1601 outputs a supplementary code 
string comprising 2 sets of extra__bit_slice and 

is extra_information_slice, i.e., 18 bits in total as shown in 
Fig. 18C of Fig. 18. By outputting such a supplementary 
code string, the head of the macroblock code string 
starts correctly from the byte border. 
[0087] Incidentally, since the supplementary code 

20 string is outputted utilizing a portion prescribed even in 
the MPEG standard, the decoder can correctly decode 
an encoded thumbnail moving image with such a code 
string as an MPEG video sequence. Further, although, 
in Fig. 18C of Fig. 18, "1111 1111" is outputted as 

25 extra_information_slice a different value is also accept- 
able. 

[0088] As described above, the processing of the step 
1 70 1 ensures that a macroblock code string constituting 
a slice 1 0 starts from a byte border position. 

30 [0089] A description will be given hereinbelow of an 
operation of the encoding work memory 1602. 
[0090] Fig. 19 shows an operation of the encoding 
work memory 1602. The processing of the step 1702 
makes the contents of the encoding work memory 1 602 

35 empty [see Fig. 19A of Fig. 19]. Meanwhile, an encod- 
ing work memory is also a portion of a computer, and 
the readout and writing access thereto are made in 
units of bytes. Thus, a bit-length effective within the byte 
is taken through the use of a variable b. As shown in Fig. 

40 19B of Fig. 1 9, the effective code string is constructed to 
be 35 bits, and hence, when 35 is divided by 8, the 
residual 3 bits are expressed using bits of a portion of 
the fifth byte in the encoding work memory 1602. 
Accordingly, the effective bit-length of the fifth byte is 

45 expressed in a manner that the variable b is set to 3. 
The section Fig. 19C of Fig. 19 shows the variation in 
the contents of the encoding work memory 1602 occur- 
ring when a 7-bit-length code string "1111 1 1 1 " is added 
to the encoding work memory 1602. 

so [0091] As described above, the code string is main- 
tainable in units of bits by using the variable b in the 
encoding work memory 1602. 

[0092] Fig. 20 is an illustration of the contents of the 
encoding work memory 1602 in the case that the-mov- 
55 ing image encoding means 1601 conducts the MPEG 
encoding processing (DCT, quantization, movement 
estimation, variable-length encoding) in the steps 1 703 
and 1 704 in Fig. 17. As shown in Fig. 20A of Fig. 20, the 
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contents of the encoding work memory 1602, when 
being in the initial condition, is vacant due to the 
processing of the step 1 702. 

[0093] When the code string obtained by the MPEG 
encoding processing for a macroblock m in the next 
step 1703 is added to the encoding work memory 1602, 
the contents of the encoding work memory 1602 
becomes as shown in Fig. 20B of Fig. 20. In Fig. 20B of 
Fig. 20, since the value of the variable b is 5, the end of 
the macroblock crosses a byte border by 5 bits. Thus, 
macroblock_stuff ing is inserted into the head portion of 
the macroblock code string so that the end of the mac- 
roblock comes into contact with the byte border. As 
shown in Fig. 20B r macroblock_stuffing is a code string 
having an ll-bit length, and whenever one 
macroblock_stuffing is inserted into the head of the 
encoding work memory 1602, the code strings already 
existing within the encoding work memory 1602 can be 
shifted by 1 1 bits in the backward direction. In the case 
of crossing the byte border by 5 bits, when one 
macroblock_stuffing is inserted thereinto, the variable b 
becomes 0, so that the end of the macroblock goes into 
a byte-aligned condition [see Fig. 20C of Fig. 20]. 
[0094] The code string undergoing the 

macroblock_stuffing insertion processing into the head 
portion of the macroblock takes a correct arrangement 
as an MPEG video sequence as shown in Fig. 52B. 
[0095] Fig. 21 shows the relationship between the 
value of the variable b before the insertion of 
macroblock_stuffing and the number of 
macroblock_stuff ing to be inserted at that time. 
[0096] As described above, the macroblock code 
string is made so that its head and end come into con- 
tact with the byte border within the encoding work mem- 
ory 1602. The moving image encoding means 1601 
outputs the macroblock code string within the encoding 
work memory 1602 as a portion of an encoded thumb- 
nail moving image code string. 
[0097] Incidentally, in the case of accomplishing a 
multi-image combination in units of macroblocks, the 
portion necessary as an encoded thumbnail moving 
image is a macroblock code string in which the head 
and end of the code string come into contact with a byte 
border. Although the slice header code string is output- 
ted in the step 1 701 , there is no need to always output 
the slice header code string into the code strings of an 
encoded thumbnail moving image. 
[0098] Furthermore, by outputting a code string 
(sequence header, GOP header, picture header and 
others) required as the MPEG video sequence, the 
encoded thumbnail moving image can be decoded as a 
single MPEG video sequence, and further reproduced 
and displayed. 

[0099] A description will be made hereinbelow of the 
processing of combining a plurality of encoded thumb- 
nail moving images into a multi-image combined mov- 
ing image. In this case, as well as the first embodiment 
let it be assumed that the moving image listing means 



209 issues a request to the moving image composing 
means 1603 for composing a multi-image combined 
moving image comprising two divisions in the vertical 
direction and two divisions in the horizontal direction 

5 shown in Fig. 7. 

[0100] Fig. 22 shows an operational flow to be con- 
ducted until the moving image composing means 1603 
combines a plurality of encoded thumbnail moving 
images into as multi-image combined moving image 

10 and outputs its MPEG code string. 

[01 01 ] In Fig. 22, in a step 2201 ( a header code string 
of each of a GOP header, a picture header and a slice 
header is outputted to allow a multi-image combined 
moving image to be decoded as an MPEG video 

is sequence, and in a step 2202, a macroblock code string 
is read out from each of encoded thumbnail moving 
images. Subsequently, in a step 2203, the macroblock 
code strings read out therefrom are outputted as a 
multi-image combined moving image code string, and in 

20 a step 2204, the value of a macroblock counter m is 
checked. 

[0102] Fig. 23 is an illustration of expressing loop 
processing of loops (L) 810. 811, 812 and 2205 in the 
operational flow of Fig. 22 in C language, where the var- 
25 iable x takes the horizontal position of the image, being 
processed, as a loop counter value, and the variable y 
takes the vertical position thereof as a loop counter 
value, while the variable n assumes the slice number of 
an encoded thumbnail moving image under processing 
30 as a loop counter value and the variable m assumes the 
macroblock number of the encoded thumbnail moving 
image, being processed, as a loop counter value. 
[01 03] Fig. 24 shows an image composing processing 
sequence for one multi-image frame in this embodi- 
35 ment. the readout of the macroblock code string of each 
of the encoded thumbnail moving images in the step 
2202 and the outputting of the multi-image combined 
moving image code string in the step 2203 are done in 
the raster scanning order (the order of numerals within 
40 parentheses in Fig, 24). That is, the macroblock code 
strings read out are successively connected in the 
raster scanning order, thereby producing a multi-image 
combined moving image code string. 
[0104] As described above, the moving image encod- 
es ing means 1601 produces macroblock code strings 
whose head and end do not cross a byte border, and 
hence, the moving image composing means 1 603 can 
create a desired multi-image combined moving image 
code string by only connecting the macroblock code 
so strings in the raster scanning order, which achieves the 
speed-up of the production of a multi -image combined 
moving image. 

[01 05] Although, in the operational flow of this embod- 
iment shown in Fig. 22, one picture in the MPEG video 
55 sequence of the multi-image combined moving image is 
constructed with only one slice, even if it is constructed 
with a plurality of slices, likewise, a desired multi-image 
combined moving image is producible in a manner of 
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only connecting the macroblock code strings in the 
raster scanning order. At this time, in order for each of 
the slice head code strings and each of the macroblock 
code strings to be correctly connected to each other at 
a byte border, the slice header code string is required to 
undergo the byte alignment by the supplementary code 
string shown in Fig. 18C of Fig. 18. 
[0106] As described above, according to this embodi- 
ment, there is provided a moving image composing sys- 
tem comprising the moving image receiving means for 
receiving moving images to be combined into a multi- 
image combined moving image, the moving image 
reducing means for reducing the received moving 
images to a predetermined size to produce thumbnail 
moving images, the moving image encoding means for 
MPEG-encoding the thumbnail moving images to pro- 
duce encoded thumbnail moving images, the encoding 
work memory to be used for when the moving image 
encoding means temporarily stores the microblock code 
strings, the accumulating means for storing and accu- 
mulating the encoded thumbnail moving images 
encoded according to MPEG, the moving image com- 
posing means for combining a plurality of encoded 
thumbnail moving images accumulated in the accumu- 
lating means to produce a multi -image combined mov- 
ing image code string, the moving image listing means 
for producing a list of a combination and order of moving 
images constituting a multi-image combined moving 
image, the moving image decoding means for decoding 
the multi-image combined moving image code string 
produced by the composition, and the moving image 
displaying means for displaying the decoded moving 
imager wherein the encoding work memory has a 
means to store and retain code strings in units of bits, 
and the moving image encoding means inserts a sup- 
plementary code string so that the head and end of the 
macroblock code string take a contacting condition with 
a byte border, and further, the moving image composing 
means arranges the macroblock code strings in the 
raster scanning order to produce a multi-image com- 
bined moving image code string for making a desired 
multi-image combined moving image. 
[0107] Whereupon, the multi-image combination for 
the moving image composition by the moving image 
composing system depends upon only arranging the 
macroblocks in order, and therefore, the speed-up of the 
multi-image combined moving image production is 
achievable, which offers great practical effects. 

(Third Embodiment) 

[0108] A description will be made hereinbelow of a 
third embodiment of this invention. In the explanation of 
the third embodiment. ^me numerals are given to 
the same parts or sections as those in the above- 
described embodiments, and the description thereof will 
be omitted for brevity. 

[0109] Fig. 25 is a block diagram showing a construc- 



tion of a moving image composing system according to 
a third embodiment of this invention. 
[01 10] In Fig. 25. numeral 2501 depicts a layout table 
for, when a multi-image raster is divided in a macroblock. 

5 size into areas, retaining multi-image layout information 
on each of the areas, numeral 2502 designates a layout 
table making means for drawing up the layout table in 
response to a request from a moving image listing 
means, numeral 2503 denotes a moving image com- 

10 posing means for combining a plurality of encoded 
thumbnail moving images with different image sizes to 
create a multi-image combined moving image code 
string, and numeral 2504 signifies a moving image list- 
ing means for drawing up a list of a combination and 

15 order of moving images organizing a multi-image com- 
bined moving image having different sizes of image 
areas. 

[0111] Fig. 26 shows an operational fbw to be con- 
ducted until the moving image composing means 2503 

20 combines encoded thumbnail moving images with dif- 
ferent image sizes into a multi-image combined moving 
image and outputs an MPEG code string thereof. 
[01 1 2] In Fig. 26, the operational flow starts at a step 
2601 to draw up a layout table as the initializing 

25 processing for the multi-image combination, then fol- 
lowed buy a step 2602 to check whether the composi- 
tion processing is conducted or not when the counter 
values assume x and y, and further followed by a step 

2603 to check whether a skip macroblock code string is 
30 outputted or not when the counter values assume x and 

y. Thereafter, the operational flow advances to a step 

2604 to output a skip slice code string, and proceeds to 
a step 2605 to read a slice code string from an encoded 
thumbnail moving image expressed with a layout table 

35 T[x, y], and further goes to a step 2606 to increment the 
loop counter value x by one. 

[0113] Fig. 27 is an illustration of an example of a 
multi-image raster composed of encoded thumbnail 
moving images with different image sizes in this embod- 

40 iment In this instance, a description will be made on the 
assumption that the moving image listing means 209 
issues a request to the moving image composing 
means 2503 for the composition of a multi-image com- 
bined moving image shown in Fig. 27. In Fig. 27, the 

45 image size of each of the encoded thumbnail moving 
images is determined by arranging macroblocks by the 
following numbers in the vertical and horizontal direc- 
tions (the size of a common macroblock is 16 pixels * 16 
pixels). 

50 

encoded thumbnail moving image A : 
(5. 4) 

encoded thumbnail moving image B : 
(2, 3) 

55 encoded thumbnail moving image C : 
(2,2) 

encoded thumbnail moving image D : 
(2. 2) 
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encoded thumbnail moving image E • 
(3,2) 

[01 1 4] Let it be assumed that, as described in the first 
embodiment, each of the encoded thumbnail moving s 
images is made in the moving image encoding means 
204 and accumulated in the accumulating means 206. 
[0115] In response to a request for the multi-image 
combination shown in Fig. 27, in the step 2601, the 
moving image composing means 2501 performs the fol- 10 
lowing initializing processing through the use of the lay- 
out table making means 2502. That is, a variable X 
representative of the number of macroblocks of the 
multi-image combined moving image in the horizontal 
direction is set to 7, while a variable Y representative of is 
the number of macroblocks thereof in the vertical direc- 
tion is set to 6. Further, in terms of each of array ele- 
ments of the layout table 2501 notated as a two- 
dimensional array T[x, y] (the first subscript represents 
a horizontal image area in the case that a multi-image 20 
raster is divided in macroblock size into areas, and the 
second subscript signifies a vertical image area in this 
case), if an area corresponding to the coordinate (x, y) 
is an area in which no moving image is displayed, a 
symbol S is given to that area, and if an area corre- 25 
sponding to the coordinate (x, y) is not the leftmost area 
of the image areas constituting a multi-image raster, a 
symbol N is given thereto, while if an area correspond- 
ing to the coordinate (x, y) is the leftmost area of the 
image areas constituting the multi-image raster, the 30 
area is taken as a symbol comprising an identification 
(ID) symbol (identifier) of the corresponding encoded 
thumbnail moving image and a vertically varying coordi- 
nate value which starts from 1 at the uppermost image 
area, thus accomplishing the initialization. 35 
[0116] Fig. 28 is an illustration of the contents of the 
layout table 2501 to be taken in the case of conducting 
the multi-image combination shown in Fig. 27. In Fig. 
28. the ID symbols used for the aforesaid encoded 
thumbnail moving images are A, B, C, D and E within 40 
the measures, and the vertically varying coordinate val- 
ues are the numerals within the measures. 
[0117] Fig. 29 is an illustration of the loop processing 
of loops 2607 and 2608 in the operational flow of Fig. 26 
expressed with a C language. 45 
[01 18] In the step 2602 of Fig. 26, if the value of T[x, 
y] corresponds to the symbol N, the moving image com- 
posing means 2501 conducts no processing. In the step 
2603, if the value of T[x, y] corresponds to the symbol S, 
the moving image composing means 2501 outputs a so 
skip slice code string. 

[01 1 9] Fig. 30 is an illustration of an example of a slice 
code string skipping by one macroblock. The moving 
image composing means 2501 outputs 6 bytes, shown 
in Fig. 30, as a code string to thereby skip one-macrob- 55 
lock area, which creates an area where no moving 
image is displayed. 

[0120] In the step 2605, the identification symbol and 
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vertically varying coordinate value of the encoded 
thumbnail moving image are obtained on the basis of 
the symbol the value of T[x, y] represents. The ID sym- 
bol of the encoded thumbnail moving image is used for 
the purpose of identifying the encoded thumbnail mov- 
ing image within the accumulating means 206, while the 
vertically varying coordinate value is used as the . 
number given to the slice code string of the encoded(^eA 
thumbnail moving image to be read. In the step 805, the \f 
rewriting processing of the displaying position informa- 
tion described in the first embodiment is conducted so 
that he slice code string read in the step 2605 is dis- 
played at the position of the coordinate (x, y). 
[0121] Through the repetition of the loops 2607 and 
2608, the moving image composing means 2501 suc- 
cessively rewrites slice_start_code and MB_AddrJnc 
of the slice code strings read from the encoded thumb- 
nail moving images expressed with T[x t y], and outputs 
the resultant code strings as a multi-image combined 
moving image code string. 

[0122] As described above, according to this embodi- 
ment, a moving image composing system comprises 
the moving image listing means for drawing up a list of a 
combination and order of multiple images made up of 
encoded thumbnail moving images with different image 
sizes, and the moving image composing means includ- 
ing the layout table for, when the multi-image raster is 
divided in macroblock size into areas, retaining multi- 
image layout information on the respective areas and 
the layout table making means for making this layout 
'table, and therefore, the moving image composing 
means determines, referring to the layout table, the slice 
code string of the encoded thumbnail moving image to 
be processed with respect to each of the coordinates on 
the multi-image raster. The layout table includes infor- 
mation on the position of each of the image areas on the 
multi-image raster corresponding to each of the coordi- 
nates on the multi-image raster and information on the 
encoded thumbnail moving image to be located at that 
image area. 

[0123] Through this process, the moving image com- 
posing means 2501 can combine a plurality of encoded 
thumbnail moving images with different image sizes to 
provide a multi-image combined moving image code 
string in which the encoded thumbnail moving images 
with different image sizes exist mixedly. This offers great 
practical effects. 

(Fourth Embodiment) 

[01 24] A description will further be made hereinbelow 
of a fourth embodiment of this invention. In the explana- 
tion of the fourth embodiment, the same numerals are 
given to the same parts or sections as those in the 
above-described embodiments, and the description 
thereof will be omitted for simplicity. 
[0125] Fig. 31 is a block diagram showing a construc- 
tion of a moving image composing system according to 
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the fourth embodiment of this invention. 
[0126] In Fig. 31 , numeral 3101 represents a moving 
image receiving means for receiving a multi-image com- 
bined moving image code string to hand it over to a 
moving image composing server 201, numeral 3102 
designates a multi-image combined moving image 
dividing means for extracting a moving image code 
string of each of image area sections from the received 
multi-image combined moving image code string, 
numeral 3103 denotes an image layout table making 
means for making an image layout table which retains 
the aforesaid multi-image combined moving image, a 
size of each of area portions of a multi-image raster and 
positional information, numeral 3104 depicts a moving 
image re- encoding means for converting the extracted 
moving image code strings into code strings (encoded 
thumbnail moving image) which are allowed to be re- 
composed within another multi-image raster, numeral 
3105 signifies a moving image composing means for 
combining the re-encoded thumbnail moving images 
and the encoded thumbnail moving images accumu- 
lated in the accumulating means 206 into a multi-image 
combined moving image code string, and numeral 3106 
indicates a moving image listing means for drawing up a 
list of a combination and order of multiple images to be 
re-constructed. 

[0127] First, a description will be made herinbelow of 
the processing of dividing code strings of image area 
portions from an inputted multi-image combined moving 
image code, string. 

[0128] Fig. 32 shows an operational flow to be con- 
ducted until the multi-image combined moving image 
dividing means 3102 extracts a moving image code 
string of each of image area portions constituting a 
multi-image faster from a multi-image combined moving 
image code string and outputs the extracted moving 
image code strings as different moving image code 
strings. 

[0129] In Fig. 32, the operational flow begins at a step 
3201 to check whether or not layout information indica- 
tive of an image configuration (layout) in that multi- 
image raster is included within a multi-image combined 
moving image data code string, and then, advances to a 
step 3202 to draw up an image layout table if the layout 
information is included therein, and further, proceeds to 
a step 3203 to read a code string corresponding to one 
frame from the multi-image combined moving image 
data code string, and still further, goes to a step 3204 to 
search slice_start_code (see Fig. 52A) for extracting 
— ^ each of the slice code strings. Subsequently, a step 
3205 is implemented to take the positional information 
on the slice undergoing the extraction, and then fol- 
lowed by a step 3206 to output to the moving image re- 
en codina means 3104 the extracted slice code string 
together with the identification symbol ID of the encoded 
thumbnail moving image, the positional information on 
the slice in the multi-image raster and the positional 
information on the slice in each of the image areas. 



After this, a step 3207 is executed to check whether all 
the slices in the code string corresponding to one frame 
are processed or not, and subsequently, a step 3208 is 
implemented to check whether all the frames of the 
5 inputted multi-image combined moving image data are 
processed or not. 

[01 30] Fig. 33 is an illustration of a format of an image 
layout table. 

[0131] In the image layout table of Fig. 33, the size of 
10 the inputted multi-image combined moving image data 
in units of macroblocks is stored in "moving image hori- 
zontal size" and "moving image vertical size", the 
number of image areas constituting a multi-image raster 
is stored in "number of images", the identification sym- 
rs bol ID of an encoded thumbnail moving image located in 
the image area m is stored in "identification symbol ID of 
image rrf , the coordinate position of the left and upper 
corner of the image area m is put in "positional informa- 
tion on image rrf\ and the size of the image area m is 
20 put in "horizontal size of image m" and "vertical size of 
image m". 

[0132] Fig. 34 is an illustration of an example of an 
image layout table made out when inputted is multi- 
image combined moving image data having an image 
25 configuration shown in Fig. 27. The following description 
will be made in the case that the inputted multi-image 
combined moving image data has an image configura- 
tion shown in Fig 27. 

[0133] The production of an image layout table is 
30 made by an image layout table making means, and in 
the step 3201, a check is made as to whether or not an 
image layout table encoded is outputted into an 
extended code area of each of header portions 
(sequence header, GOP header, picture header) of the 
35 multi -image combined moving image data code string 
or a user data area thereof. If outputted, in the step 
3202, that code string is read, thereby drawing up the 
image layout table. 

[0134] Besides, for outputting an image layout table 
40 into a multi-image combined moving image code string, 
the moving image composing system for producing and 
outputting a multi-image combined moving image code 
string performs processing to output an image layout 
table into a code string. For instance, the moving image 
45 composing means 2503 in the third embodiment con- 
verts the layout table shown in Fig. 28 into the image 
layout table shown in Fig. 34, and encodes and outputs 
the image layout table. 

[0135] The extraction of each of slice code strings is 
so accomplished by searching slice_start_code represent- 
ing the head of the slice. The positional information 
(information on a position in a multi-image raster) on the 
extracted slice code string is obtained as follows in the 
step 3205. 

55 

horizontal position = value of MB_Addr_lnc of head 
macroblock 

vertical position = value of fourth byte of 
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slice_start_code 

[0136] The multi-image combined moving image 
dividing means 3102 obtains the identification symbol 
ID of an encoded thumbnail moving image and the posi- 
tional information on a slice in each of image areas on 
the basis of the image layout table and the positional 
information on slice code strings in a multi-image faster, 
and outputs them to the moving image re-encoding 
means 3103. 

[0137] Fig. 35 shows identification symbols ID of 
encoded thumbnail moving images and positional infor- 
mation on slices in each of image areas, obtained from 
the examples of the image layout table and positional 
information shown in 

[0138] Fig. 34. In Fig. 35, they are arranged in the 
order processed in the operational flow shown in Fig. 
32. 

[0139] A description will be given hereinbelow of the 
processing of converting a slice code string, divisionally 
extracted by the multi-image combined moving image 
dividing means 3102, into a code string to be re-com- 
bined in a different multi -image raster 
[0140] Fig. 36 is an illustration of an example in which 
the moving image re-encoding means 3104 converts a 
slice code string into a code string to be re-combined in 
a different multi-image raster on the basis of a slice 
code string, positional information on a multi-image 
raster and positional information on each image area 
handed over by the multi-image combined moving 
image dividing means 3102. In this instance, the values 
the moving image re-encoding means 3104 receives 
from the multi-image combined moving image dividing 
means 31 02 are as follows. 

slice code string : code strings shown in Fig. 36A 
and Fig. 20B and/or Fig. 36B of Fig. 36 
positional information on multi-image raster : (5, 6) 
positional information on image area : (1 f 2) 
identification symbol ID : E 

[0141 ] In other words, Fig. 36 shows the conversion of 
the second slice code string of the encoded thumbnail 
moving image E disposed at the coordinate (5, 6) in the 
multi-image combined moving image shown in Fig. 27. 
[0142] In Fig. 36, the section Fig. 36A shows a slice 
code string handed over by the multi-image combined 
moving image dividing means 3102, while the section 
Fig. 36B of Fig. 36 illustrates a detail construction in 
units of bits. Since the vertical position information on 
the slice code string is stored in the fourth byte of 
slice_start_code, that is, buf[3], the substitution 
processing for buf[3] = 2 is made in order to set the 
value of buf[3] to the vertical position "2" indicated with 
the positional information on the image area. 
[0143] Furthermore, although the horizontal position 
of the slice code string is required to be rewritten from 5 
to 1, because of being a variable-length code bit 



MB_AddrJnc portion, if expressing as a variable- length 
code, the horizontal position 5 is "0010", and the hori- 
zontal position 1 is "1" in variable-length code. Accord- 
ingly, although 3 bits are left in the MB_AddrJnc portion 

s when the horizontal position is rewritten from 5 to 1, 
these bits are absorbed in a manner that 3 padding 
bytes are inserted as shown in Fig. 36C of Fig. 36. Fur- 
ther, as shown in Figs. 36C and 36D of Fig. 36, the 
rewriting operation is made to set the horizontal position 

io to1. 

[0144] The slice code string that the slice position 
information is rewritten in this way is sent as a code 
string of the encoded thumbnail moving image E, which 
can be combined in units of slices, to the accumulating 

is means 206 or directly to the moving image composing 
means 3104 to be used for the composition of a multi - 
image combined moving image. 
[0145] As described above, according to this embodi- 
ment, there is provided a moving image composing sys- 

20 tern comprising the moving image receiving means for 
receiving a multi-image combined moving image, the 
multi-image combined moving image dividing means for 
extracting a slice code string of each of image area por- 
tions from the received multi-image combined moving 

25 image code string, the image layout table making 
means for drawing up an image layout table retaining a 
size of each of the image areas of a multi-image raster 
and positional information in corresponding relation, the 
moving image re-encoding means for converting the 

30 slice code string into a slice code string which can be 
combined in a different multi-image raster, and the mov- 
ing image composing means for recombining the re- 
encoded slice code strings to produce a multi-image 
combined moving image code string with a different 

35 image configuration, wherein the moving image dividing 
means fetches positional information from the header 
portion of the extracted slice code string and refers to 
the image layout table on the basis of this positional 
information to take out the identification symbol ID of the 

40 corresponding encoded thumbnail moving image and 
the positional information on each of the image areas 
thereof, and receiving this taken results, the moving 
image re-encoding means converts the positional infor- 
mation on the slice code string from information on the 

45 multi- image raster into information on each image area 
so that the slice code string is converted into a code 
string which is allowed to be combined in a different 
multi-image raster. 

[0146] Through the above-described process, it is 
so possible to re-construct a multi-image combined moving 
image into a multi-image combined moving image with 
a different image configuration. Particularly, since the 
moving image re-encoding means 3103 outputs a slice 
code string identical in format with that of an encoded 
55 thumbnail moving image to be outputted from the mov- 
ing image encoding means 204 described in the first 
embodiment, the slice code string outputted can be re- 
combined in units of slices in the moving image com- 
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posing means, which offers a great effect. 
(Fifth Embodiment) 

[0147] A description will be made hereinbelow of a 
fifth embodiment of this invention. In the explanation of 
the fifth embodiment, the same numerals are given to 
the same parts or sections as those in the above- 
described embodiments, and the description thereof will 
be omitted for simplicity. 

[0148] Fig. 37 is a block diagram showing a construc- 
tion of a moving image composing system according to 
the fifth embodiment of this invention. 
[0149] In Fig. 37, numeral 4101 designates an image 
encoding means for conducting the encoding process- 
ing of an inputted image on the basis of an image 
encoding format, numeral 4102 denotes an accumulat- 
ing means for storing and accumulating the encoded 
images, numeral 4103 depicts an image composing 
means for receiving at least one encoded image to pro- 
duce a multi-image combined image stream, and 
numeral 4104 signifies an image decoding means for 
decoding the multi-image combined image stream to 
produce and display a multi-image combined image. 
[0150] Secondly, a description will be give hereinbe- 
low of an operation of the moving image composing 
system according to this embodiment. 
[0151] In this embodiment, since an example of an 
MPEG format is used as the inputted image encoding 
format, the image encoding means 4101 is the same as 
the moving image encoding means 204 according to the 
above-described first embodiment, and hence, its inter- 
nal operation is the same as that described in the first 
embodiment. That is, the inputted image is a thumbnail 
image and the encoded moving image is an encoded 
thumbnail moving image. 

[0152] Accordingly, the image encoding means 4101 
outputs, to the accumulating means 4102, an image 
compression-encoded according to an MPEG format as 
described in the first embodiment. 
[0153] The accumulating means 4102 is the same as 
that of the accumulating means 205 in the first embodi- 
ment, and its internal operation is as described. In addi- 
tion, this accumulating means 4102 stores and 
accumulates the encoded images and, at this time, allo- 
cates a singly distinguishable image ID for each of the 
encoded images for the accumulation management. In 
Fig. 10A, 5 encoded images are accumulated, and the 
image IDs A, B, C, D and E are given to the encoded 
images, respectively, for the accumulation manage- 
ment. A given encoded image can be fetched from the 
accumulating means on the basis of the image ID des- 
ignated. Each of the encoded images has a construc- 
tion equal to that of the encoded thumbnail image 
shown in Fig. 6. 

[0154] The image composing means 4103 takes out 
at least one encoded image from the accumulating 
means 4102 and conducts the image combining 



processing thereof to produce and outputs a multi- 
image combined image stream. Incidentally, in this 
embodiment, the image composing means is identical 
in structure with the moving image composing means 
s 207 in the first embodiment, and its internal operation is 
as described above. 

[01 55] A description of this embodiment will be made 
in the case of producing a multi-image combined image 
shown in Fig. 38. That is, the image composing means 

10 4103 composes a multi-image combined image in a 
manner that each of the encoded images stored in the 
accumulating means 4102 is displayed in each of a total 
of four areas obtained by dividing a multi-image raster 
into two in the vertical direction and two in the horizontal 

is direction. 

[01 56] For displaying the multi-image combined image 
shown in Fig 38, the multi-image combined image 
stream to be outputted from the image composing 
means 4103 must be a code string sequence, for exam- 

20 pie, shown in Fig. 39. For this reason, the image com- 
posing means 4103 conducts the code string producing 
processing shown in Fig. 39 through the steps shown in 
Fig. 3. A multi-image combined image stream produced 
through this code string producing processing shows a 

25 correct code string as the MPEG format, so that the 
MPEG reproduction becomes possible in the image 
decoding means 4104. 

[0157] The above-described construction of the mov- 
ing image composing system allows the information 

30 value to be changed without the need for an extremely 
large amount of processing such as the bit shift opera- 
tion in connection with the code length variation of the 
information occurring due to the change of the informa- 
tion value defined as a variable-length code, which con- 

35 tributes to the speed-up of the multi-image combined 
image stream. 

[01 58] Furthermore, in this case, since the code string 
sequence of the multi-image combined image stream 
produced by the image composing means 4103 is cor- 

40 rect as the MPEG format, the reproduction and display 
become feasible in the image decoding means 4104 for 
decoding the MPEG code string, or an image decoding 
unit (generally called MPEG decoder). 
[01 59] In this embodiment, the image to be inputted to 

45 the image encoding means can be either a static picture 
or a moving image. In the case of the still picture, the 
processing in this embodiment is equivalent to the case 
that the inputted image is only one frame in the portion 
to be processed on frame levels (see the operation of 

so the image encoding means in the first embodiment). 
[0160] Furthermore, it is also acceptable that the 
inputted image is encoded in advance through the use 
of some image encoding format, in this case, the image 
encoding means 4101 performs the converting process- 

55 ing of the image encoding format wrtn respect to the 
inputted image. 

[0161] Still further, in this embodiment although the 
MPEG format is used as the image encoding format for 
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the inputted image and the multi-image combined 
image stream, in terms of the image encoding format 
standards, this invention is applicable to all the image 
encoding formats each having information defined as a 
variable-length code. 5 
[0162] Still further, in the description of this embodi- 
ment, although the basic operation unit length in the 
image composing means for rewriting the information 
defined as a variable-length code is 8 bits (which are 
currently and generally referred to as one byte), it is also 10 
possible to employ an arbitrary basic unit length not less 
than 1. 

[01 63] Still further, in the example of this embodiment, 
although the horizontal position information is used as 
the information to be rewritten, in terms of the image is 
encoding format standards, it is also possible to use any 
information defined as a variable-length code. In addi- 
tion, it is also acceptable that a plurality of information 
are changed at a time. 

[0164] Moreover, in this embodiment, although a com- 20 
bination of extra_bit_slice and extra_jnformation_slice 
is used as an extended code string to be used when 
supplying wanted bits, it is also appropriate that a por- 
tion defined as a code string which does not affect the 
image in the image encoding format or which hardly has 25 
influence thereon is outputted in advance into the 
encoded image. 

(Sixth "Embodiment) 

30 

[0165] A description will be given hereinbelow of a 
sixth embodiment of this invention. In the explanation of 
the sixth embodiment, the same numerals are given to 
the same parts or sections as those in the above- 
described embodiments, and the description thereof will 35 
be omitted for simplicity. 

[0166] Fig. 40 is a block diagram showing a construc- 
tion of a moving image composing system according to 
this embodiment 

[0167] In Fig. 40, numerals 4401 and 4402 denote 40 
delay means for temporarily or long accumulating input- 
ted data, retaining it or conducting some processing 
therefor, and then outputting it, numeral 4103 depicts a 
composition server for accumulating and managing 
encoded images, and numeral 4104 signifies a repro- 45 
duction terminal for conducting the processing for the 
composition of a multi-image combined image stream 
and further for reproducing and displaying the resultant. 
[0168]: A description will be made hereinbelow of an 
operation of the moving image composing system so 
according to this embodiment. 

[0169] The delay means 4401 and 4402 output data 
inputted thereto in a delayed condition through the use 
of a storage means including a magnetic disk and a 
memory device, or adjusts the delay time resulting from 55 
the frame thinning processing. 
[0170] In this case, the delay means 4401 (which will 
be referred hereinafter to as a delay means 1) delays a 



code string of an encoded image from an image encod- 
ing means by a constant time and then outputs it. This 
delay time can be designated from the external of the 
delay means 1 , and for example, the reproduction termi- 
nal 4104 can give an instruction for a delay of 5 sec- 
onds. In addition, the delay time is changeable. For 
example, it is also possible to perform a catch-up repro- 
duction in a manner that the delay time is gradually 
reduced from the initial state in which the delay time is 
10 seconds so that no delay time takes place (the delay 
time reaches zero) 1 minute after the initial state, and 
this processing is also included in the processing of the 
delay means 1 . 

[0171] On the other hand, the delay means 4402 
(which will be referred hereinafter to as a delay means 
2) outputs a code string at every four frames in connec- 
tion with the encoded images fetched from the accumu- 
lating means 4102 (that is, outputs code strings of 
frames whose numbers are 1, 5, 9, 13 • • •). Thus, the 
encoded image outputted from the delay means 2 is an 
image whose reproduction rate is increased to four 
times. 

[0172] In this embodiment, the image composing 
means 4103 receives encoded images through four 
paths as shown in Fig. 41 . 

[0173] In Fig. 41, the image encoding means 4101 
receives a real-time live picture from a camera as an 
inputted image and conducts the MPEG encoding 
processing thereof to output an encoded image whose 
image ID is X. 

[0174] Furthermore, the delay means 1 conducts the 
change of the image ID in addition to performing the 
aforesaid delay processing, and in this instance, 
changes it to X'. Further, an encoded image whose 
image ID is B is taken out from the accumulating means 
4102. The delay means 2 changes the image ID to B' in 
addition to conducting the aforesaid delay processing. 
[0175] Accordingly, the image composing means 
4103 performs the combining processing of four 
encoded images whose image IDs are X, X', B and B\ 
[0176] As a construction of the moving image com- 
posing system, it is also appropriate that, as shown in 
Fig. 40, the image composing means 41 03 is placed on 
the reproduction terminal 4404 side. At this time, the 
internal processing of the image composing means 
4103 is the same as that in the first embodiment, and 
the image decoding means 4104 decodes and repro- 
duces the multi-image combined image stream com- 
posed by the image composing means 4103. 
[0177] Fig. 42 is an illustration of an example of a 
multi-image combined image reproduced according to 
this embodiment. Fig. 42 shows a reproduced example 
in the case that various kinds of images such as a live 
picture, a delayed picture of the live picture, an accumu- 
lated image and a speed-changed image of the accu- 
mulated image are combined in image areas into a 
multi-image combined image. 

[01 78] With the above-described construction of the 
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moving image composing system, a multi-image com- 
bined image such as a combination of a real-time live 
picture such as an image coming as an inputted image 
from a camera and an accumulated image can be dis- 
played, and the delay reproduction of a live picture por- 
tion by the delay means is possible while the 
reproduction and display of a multi-image combined 
image undergoing the adjustment of the delay time due 
to the frame thinning processing are feasible. 

(Seventh Embodiment) 

[0179] A description will be made hereinbelow of a 
seventh embodiment of this invention. In the explana- 
tion of the seventh embodiment, the same numerals are 
given to the same parts or sections as those in the 
above-described embodiments, and the description 
thereof will be omitted for simplicity. 
[0180] Fig. 43 is a block diagram showing a construc- 
tion of a moving image composing system according to 
this embodiment. 

[0181] In Fig. 43, numeral 4701 represents a layout 
table for retaining an arrangement of areas as image 
layout information in the case that an image obtained by 
the reproduction of a multi-image combined image is 
divided into more than one areas, and numeral 4702 
denotes a layout table changing means for changing the 
value of multi-image layout information within the layout 
table 

[0182] A description will be made hereinbelow of an 
operation of the moving image composing system 
according to this embodiment. 

[0183] In the explanation of this embodiment, the 
same parts as those in the third embodiment will be 
omitted from the description. That is, a thumbnail image 
is used as an inputted image while an encoded thumb- 
nail moving image is employed as an encoded image. 
Further, the layout table 4701 is identical with the layout 
table in the third embodiment, and its structure is the 
same as described. Still further, the image composing 
means 4103 is the same as the moving image compos- 
ing means 2503 in the third embodiment, and its inter- 
nal operation is the same as described. 
[0184] Fig. 44 shows an image structure of a multi- 
image combined image before and after the alternation 
of the contents of the layout table 4701 by the layout 
table changing means 4702. 

[0185] With this embodiment, as shown in Fig. 44, the 
image composing means 4103 is capable of producing 
multi-image combined image streams fully different in 
image structure from each other. Fig. 45 is an illustration 
of processing for combination of encoded images by the 
image composing means 4103. 
[0186] In Fig. 45. in a step 4901. the initializing 
processing is conducted like the step 2601 in the third 
embodiment to draw up a layout table having the image 
arrangement shown in the upper section of Fig. 44 as 
the initialized state, and then, in a step 4902, the image 
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composing means 4103 refers to the layout table. 
[0187] At the time of the start of the processing on 
frame levels in the multi-image combined image stream 
producing processing, the image composing means 

5 4103 refers to the layout table at that time (step 4902) to 
perform the composing processing (step 803 to step 
809 in Fig. 45) in accordance with the multi-image lay- 
out information within the layout table. Whereupon, the 
image structure change of the multi-image combined 

10 image by the layout table changing means 4702 is done 
on frame levels. 

[0188] Thus, the above-described construction of the 
moving image composing system allows the dynamic 
change of the disposition of areas in a mufti-image com- 
rs bined image, the sizes of the areas, the encoded 
images to be displayed in the areas and others. 

(Eighth Embodiment) 

20 [0189] A description will be made hereinbelow of an 
eighth embodiment of this invention. In the explanation 
of the seventh embodiment, the same numerals are 
given to the same parts or sections as those in the 
above-described embodiments, and the description 

25 thereof will be omitted for simplicity. 

[01 90] Fig. 46 is a block diagram showing a construc- 
tion of a moving image composing system according to 
this embodiment. 

[0191] In Fig. 46, numeral 5001 designates an image 
30 encoding means for performing the encoding process- 
ing of an inputted image on the basis of an image 
encoding format to output an encoded image, and 
numeral 5002 an image composing means for receiving 
at least one encoded image to produce a multi-image 
35 combined image stream. 

[0192] Secondly, a description will be taken hereinbe- 
low of an operation of the moving image composing 
system according to this embodiment. 
[0193] The description of this embodiment begins at 
40 the inputted image encoding processing by the image 
encoding means 5001 . 

[0194] In this embodiment, the image encoding 
means 5001 conducts the encoding processing of an 
image inputted thereto according to an MPEG format, 

45 with the encoded image being accumulated in the accu- 
mulating means 4102. The MPEG encoding operational 
flow in the image encoding means 5001 is the same as 
that in the first embodiment described with reference to 
Fig. 3. Incidentally, the inputted image in this embodi- 

so merit is treatable as being the same as the thumbnail 
moving image in the description using Fig. 3. 
[01 95] Fig. 47 is an illustration of an example of a code 
string in the case that the image encoding means 5001 
MPEG-encodes an inputted image toe output an 

55 encoded image in the steps 303. 305. 306 and 308. 
[0196] In Fig. 47, the sections (a), (b), (c) and (d) cor- 
respond to (a), (c), (d) and (e) in Fig. 5. respectively. 
Thus, the code string of the encoded image in this 
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embodiment is the same as that produced by a common 
MPEG encoding means or MPEG encoding unit (gener- 
ally referred to as an MPEG encoder). Accordingly, a 
common MPEG encoding means can be used as the 
image encoding means 5001 in this embodiment. 
[0197] The encoded image outputted from the image 
encoding means 5001 is accumulated and managed by 
the accumulating means 4102. Fig. 48 is an illustration 
of encoded images accumulated in the accumulating 
means 4102. In Fig. 48, the encoded images respec- 
tively identified with the image IDs A, B, C, D and E are 
accumulated in the accumulating means 4102. Further, 
each of the encoded images is horizontally divided into 
two slices each of which is made up of three macrob- 
locks. 

[01 98] Secondly, a description will be given hereinbe- 
low of rewriting processing of an information value 
defined with a variable length of an encoded image in 
this embodiment. The description of this embodiment 
will be made, for example, in the case of composing 
such a multi-image combined image as shown in Fig. 
38. 

[0199] The multi-image combined image composing 
processing is conducted by the image composing 
means 5002, and its operational flow is the same as that 
of the first embodiment described with reference to Fig. 
8. Incidentally, the encoded image in this embodiment is 
treatable as being identical with the encoded thumbnail 
moving image in Fig. 8. 

[0200] Figs. 49A and 49B is an illustration of readout 
processing of a slice code string the image composing 
means 5002 conducts in the step 804. The slice code 
string read out is stored in a buffer whose head address 
is pointed to by a variable Bui 

[0201] The image composing means 5002 conducts 
the rewriting processing of the slice displaying position 
information as follows in the step 805. The description 
of the first embodiment made with reference to Fig. 1 1 
applies to the rewriting of the vertical position. 
[0202] The rewriting processing of the horizontal posi- 
tion information defined as a variable-length code in the 
step 805 will be described in the case that the value of 
the horizontal position varies from 1 to 4. In the condi- 
tion stored in the accumulating means 4102, as shown 
in Fig. 47, the horizontal position information value of 
the encoded image is 1 , and its code string is "1" being 
one-bit length. For rewriting this to the horizontal posi- 
tion 4, when being expressed with a variable-length 
code, the horizontal position 4 becomes "001 V of four- 
bit length. Accordingly, the lack of the code string of 3 
bits occurs. 

[0203] Referring to Fig. 50, a description will be given 
hereinbelow of a method of supplying a wanted code 
string and of rewriting the horizontal position information 
value defined as a variable-length code at a high speed. 
[0204] In Fig. 50, Fig. 50A shows a state where the 
code string of a slice 1 A is read into a buffer in the step 
804, while Fig. SOB is a detailed illustration of the vicinity 



of a slice header at that time. 

[0205] First, the image composing means 5002 
rewrites the buf[4] portion. In this embodiment, the 
rewriting is done as buf [4] = buf[4] | 0x07. 
5 [0206] Following this, 5 bytes, from buf [0] to buf[4], 
are outputted as a code string of a multi-image com- 
bined image stream. 

[0207] Subsequently, as 6-byte insertion processing, 
6 bytes being Oxff are outputted as a code string of the 

io same multi-image combined image stream. 

[0208] Finally, after the rewriting processing of the hor- 
izontal position information is done as buf[10] = 0xc6 + 
(buf[4] & 0x01), all the slice code strings succeeding 
buf [10] are outputted as code strings of the multi-image 

is combined image stream. 

[0209] Incidentally, strictly speaking, since the value of 
buf[4] is referred to at the rewriting processing of the 
horizontal position information, there is a need to tem- 
porarily store the value of buf[4] in a variable at the first 

20 rewriting of buf[4]. . 

[0210] Through the above-described processing, the 
rewriting of the information value defined as a variable- 
length code can be completed by only making simple 
calculation several times. Accordingly, coping with the 

25 code length variation of the information occurring due to 
the variation of the information value defined as a varia- 
ble-length code, the change of the information value is 
made possible without requiring an extremely large 
amount of processing such as bit shift operation, which 

30 greatly contributes to the speed-up of the multi-image 
combined image producing processing. 
[0211] In addition, since the image encoding means 
for producing an encoded image to be inputted to the 
image composing means is not required to conduct par- 

35 ticular processing for realizing the aforesaid speed-up, it 
is possible to use a general image encoding means 
which performs the encoding processing on the basis of 
the aforesaid image encoding format, and further to pro- 
mote the speed-up of the combining processing among 

40 common encoded images based upon the aforesaid 
image encoding format. 

[0212] As described above, according to this inven- 
tion, in the moving image composition or combination, 
because of taking the order of the reduction of moving 

45 images taken, the encoding, (the accumulation), the 
composition and the decoding, as compared with the 
prior method in which the composition (the arrangement 
for display) is made after the decoding so that the 
decoding processing is needed by the images to be 

so combined, when the decoding processing is conducted 
once after the composition, the decoding is completed, 
and therefore, naturally the time required for the compo- 
sition processing is reducible to a minimum, and further, 
since the encoding is done to facilitate the change of the 

55 positional information at the composition in considera- 
tion of the case that the location of the reduced and 
encoded image within the image after the composition 
is unknown before the combination, it is possible to per- 
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form a variety of image composition (the disposition and 
display of a reduced image before the composition at an 
arbitrary position on a multi-image combined image 
after the composition). 

[0213] It should be understood that the foregoing s 
relates to only preferred embodiments of the present 
invention, and that it is intended to cover all changes 
and modifications of the embodiments of the invention 
herein used for the purpose of the disclosure, which do 
not constitute departures from the spirit and scope of 10 
the invention. 

Claims 

1 . A moving image composing system comprising: is 

moving image receiving means for receiving a 
moving image; 

moving image reducing means for reducing 
said moving image to produce a reduced mov- 20 
ing image; 

moving image encoding means for encoding 
said reduced moving image to produce an 
encoded moving image; 
: accumulating means for storing and accumu- 25 3. 
lating said encoded moving image; 
moving image composing means for combining 
at least one encoded moving image accumu- 

- lated in said accumulating means into a multi- 
image combined moving image; 30 
moving image decoding means for decoding 
said multi-image combined moving image; and 
moving image displaying means for displaying 4. 
the decoded moving image, 

wherein said movirjg image encoding means 35 
: conducts the encoding on the basis of posi- 
tional information said reduced moving image 
has to facilitate a change of said positional 
information in consideration with a case that a 
, variation of said positional information occurs 40 
due to a disposition of said reduced moving 

- image at the moving image combination, and 5. 
said moving image composing means changes 

said positional information to positional infor- 
mation required for its combining processing 45 
when a need for the change of the positional 
information said reduced moving image has 
takes place. 

2. A moving image composing system comprising: so 

moving image receiving means for receiving a 
moving image; 

moving image reducing means for reducing 
said moving image to produce a thumbnail 55 
moving image; 

moving image encoding means for encoding 
said thumbnail moving image according to an 



MPEG standard to produce an encoded 
thumbnail moving image; 
accumulating means for storing and accumu- 
lating said encoded thumbnail moving image; 
moving image composing means for combining 
one or more encoded moving images accumu- 
lated in said accumulating means into a multi- 
image combined moving image; 
moving image decoding means for decoding 
said multi-image combined moving image; and 
moving image displaying means for displaying 
the decoded moving image, ' 
wherein said moving image encoding outputs a 
supplementary code string for making up for a 
wanted bit in consideration with a case that the 
number of bits of an MB_Addr_ Inc varies due 
to a change of a horizontal coordinate of a 
slice, and 

said moving image composing means rewrites 
a value of MB_Addr_lnc by supplying the 
wanted bit from said supplementary code 
string and produces a multi-image combined 
moving image code string. 

A moving image composing system as defined in 
claim 2. wherein said moving image encoding 
means and said moving image composing means 
utilize, as said supplementary code string for mak- 
ing up for the wanted bit, extra_bit_slice and 
extraJnformation_slice of a slice header section in 
an MPEG video sequence. 

A moving image composing system as defined in 
claim 3, wherein said moving image encoding 
means outputs 8 sets of extra_bit_slice and 
extra_information_slice as said supplementary 
code string, and said moving image composing 
means supplies an arbitrary number of wanted bits 
from said supplementary code string, and produces 
a macroblock_escape code string. 

A moving image composing system comprising: 

moving image receiving means for receiving a 
moving image; 

moving image reducing means for reducing 
said moving image to produce a thumbnail 
moving image; 

moving image encoding means for encoding 
said thumbnail moving image according to an 
MPEG standard to produce an encoded 
thumbnail moving image; 
accumulating means for storing and accumu- 
lating said encoded thumbnail moving image; 
moving image composing means for combining 
one or more encoded moving images accumu- 
lated in said accumulating means into a multi- 
image combined moving image; 
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moving image decoding means for decoding 
said multi-image combined moving image; and 
moving image displaying means for displaying 
the decoded moving image, 
wherein said moving image encoding means s 
inserts a supplementary code string so that a 
head and end of each of macroblock code 
strings come into contact with a byte border, 
and 

said moving image composing means succes- w 
sively arranges said macroblock code strings in 
a raster scanning order to produce a desired 
multi-image combined moving image code 
string. 

75 

6. A moving image composing system as defined in 
claim 5, wherein said moving image encoding 
means and said moving image composing means 
utilize, as said supplementary code string to be 
used for making said head of said macroblock code 20 
string come into contact with said byte border, 
extra _bit_sl ice and extrajnformation_slice of a 
slice header section in an MPEG video sequence. 

7. A moving image composing system as defined in 25 
claim 5, wherein said moving image encoding 
means and said moving image composing means 
inserts, as said supplementary code string to be 
used for making said end of said macroblock code 
string come into contact with said byte border, 30 
macrob!ock_stuffing in an MPEG video sequence 
into a head portion of said macroblock code string. 

8. A moving image composing system as defined in 
claim 2, wherein said moving image composing 35 
means includes a layout table for, when a multi- 
image raster is divided in macroblock size into 
areas, retaining multi-image layout information on 
each of said areas, and layout table making means 

for drawing up said layout table, while said moving 40 
image composing means refers to said layout table 
to determine an encoded thumbnail moving image 
code string to be outputted as said multi-image 
combined moving image code string. 

45 

9. A moving image composing system comprising: 



moving image re-encoding means for convert- 
ing said slice code string into a re-encoded 
slice code string which is allowed to be re-com- 
bined within a different multi-image raster; 
accumulating means for storing and accumu- 
lating encoded thumbnail images; 
moving image composing means for re-com- 
bining said re-encoded slice code strings to 
produce a multi-image combined moving 
image code string providing a different image 
construction; 

moving image listing means for drawing up a 
list of a combination and order of moving 
images constituting a multi-image combined 
moving image; 

moving image decoding means for decoding 
said multi-image combined moving image; and 
moving image displaying means for displaying 
the decoded moving image, 
wherein said multi-image combined moving 
image dividing means fetches an identification 
code of an encoded thumbnail moving image, 
positional information on said encoded thumb- 
nail moving image in a multi-image raster and 
positional information on said encoded thumb- 
nail moving image in each of said image areas, 
which are required for re-encoding the 
extracted slice code string, from said layout 
table, and 

said moving image re-encoding means con- 
verts said slice code string into said code string 
to be re-combined within said different multi- 
image raster on the basis of the fetched results 
to produce said multi-image combined moving 
image different in image construction from the 
first-mentioned multi-image combined moving 
image. 

10. A moving image composing system as defined in 
claim 9, wherein said moving image listing means 
selectively draws up a list of moving images from 
said encoded thumbnail moving images accumu- 
lated in said accumulating means and moving 
images composed of the re-encoded slice code 
strings, and said moving image composing means 
obtains a code string from said encoded thumbnail 
moving image code strings accumulated in said 
accumulating means and the re-encoded slice code 
strings on the basis of said moving image list drawn 
up by said moving image listing means. 

1 1. A moving image composing system comprising: 

image encoding means for encoding an input- 
ted image on the basis of an image encoding 
format and for outputting an encoded image; 
and 

image composing means for receiving said one 



moving image receiving means for receiving a 
multi-image combined moving image; 
multi-image combined moving image dividing so 
means for extracting a slice code string of each 
of image area sections from a code string of 
said multi-image combined moving image; 
image layout table making means for drawing 
up an image layout table which retains a size of ss 
each of image areas in a multi-image raster 
and positional information in a corresponding 
relation; 
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or more encoded images to produce a multi- 
image combined image stream, 
wherein said image encoding means outputs 
an extended code string, defined as a code 
string which does not affect an image in said 
image encoding format, into said encoded 
image in advance in consideration with a case 
that a change of an information value defined 
as a variable-length code in said image encod- 
ing format takes place, and 
said image composing means changes said 
information value of said encoded image to a 
different value through the use of said 
extended code string for producing said multi- 
image combined image stream. 

12. A moving image composing system as defined in 
claim 11, wherein said image composing means 
rewrites said information value of each of said 
encoded image inputted to output said multi-image 
combined image stream in which a code string of 
each of said encoded images is a correct informa- 
tion value as said image encoding format and a 
code string sequence of each of said encoded 
images is a correct sequence as said image encod- 
ing format. 

13. A moving image composing system as defined in 
claim 11, wherein, when receiving only one 
encoded image, said image composing means 
does not produce a multi-image combined image 
but changes said information value defined as a 
variable-length code in said encoded image and 
outputs an encoded image after the change of said 
information value. 

14. A moving image composing system as defined in 
claim 11, wherein, in changing said information 
value defined as a variable-length code, said image 
composing means, when a basic operation unit 
length of said image composing means is taken to 
be N bits, takes out a bit code string corresponding 
an arbitrary multiple of N from said extended code 
string to adjust a bit string corresponding to a varia- 
tion of the information code length occurring due to 
the change of said information value. 

15. A moving image composing system as defined in 
claim 14, wherein said operation unit length is one 
of 4 bits, 8 bits forming 1 byte, 16 bits, 32 bits and 
64 bits which are operation unit lengths of a central 
processing unit of a general-purpose computer, 
respectively. 

16. A moving image composing system as defined in 
claim 11, further comprising delay means for out- 
putting inputted data after said inputted data tempo- 
rarily or long undergoes processing including an 



inputted data accumulation and an inputted data 
retention, wherein said image composing means 
receives said encoded image produced through 
said delay means. 

5 

17. A moving image composing system as defined in 
claim 16, wherein a process in which said delay 
means is not placed exists in a process on one of 
an input side of said image encoding means, a 

w process between said image encoding means and 
said image composing means and both the proc- 
esses, and said image composing means receives 
one or more encoded images produced without a 
single passage through said delay means. 

15 

18. A moving image composing system as defined in 
claim 1 1 , further comprising a layout table for, when 
a reproduced image of a multi-image combined 
image to be outputted is divided into one or more 

20 areas, retaining, as multi-image layout information, 
a construction of said areas, with said layout table 
retaining, as said multi -image layout information, at 
least positional information on disposition of each of 
said areas in a multi-image combined image and 

25 size information of each of said areas, and with said 
image composing means changing an information 
value defined as a variable-length code and an 
information value defined as a fixed-length code to 
values retained in said layout table in producing 

30 said multi-image combined image stream. 

19. A moving image composing system as defined in 
claim 18, wherein said encoded images.are distin- 
guishable by single identifiers, and said encoded 

35 images having arbitrary identifiers are obtainable 
through the use of said identifiers, and said layout 
table further retains said identifiers of said encoded 
images to be displayed in said areas as area image 
information in said multi-layout information, and 

40 said image composing means refers to said area 
image information to select said encoded image to 
be displayed in said area from the inputted images 
or to receive it as one of the inputted encoded 
images to. change said information value of said 

45 encoded image. 

20. A moving image composing system as defined in 
claim 18, further comprising layout table changing 
means for changing a value of said multi-image lay- 
so out information in said layout table, with said layout 

table changing means changing said value of said 
multi-image layout information at an arbitrary tim- 
ing, and with said image composing means refers 
to said multi-image layout information value 
55 changed to dynamically change a multi-image lay- 
out of said multi-image combined image to be out- 
putted. 
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21. A moving image composing system as defined in 
claim 1 1 , wherein said image encoding format is for 
a static image, and said encoded image is a static 
image. 

22. A moving image composing system as defined in 
claim 11, wherein said image encoding format 
relates to moving images, and said encoded image 
is a moving image. 

23. A moving image composing system as defined in 
claim 1 1 , wherein, in said image composing means, 
said encoded image to be inputted and said multi- 
image combined image stream to be outputted are 
obtained by the encoding using image encoding for- 
mats different from each other. 

24. A moving image composing system comprising an 
image composing means for receiving at least one 
encoded image, produced by encoding processing 
based upon an image encoding format to produce a 
multi-image combined image stream, wherein, 
when there is a need to change information value 
defined as a variable-length code in said image 
encoding format, said image composing means uti- 
lizes an extended code string defined as a code 
string which does not affect an image in said image 
encoding format to output said extended code 
string into said multi-image combined image for 
changing said information value in said encoded 
image to a different value. 

25. A moving image composing system as defined in 
claim 24, wherein said image composing means 
rewrites said information value of each of said 
encoded image inputted to output said multi-image 
combined image stream in which a code string of 
each of said encoded images is a correct informa- 
tion value as said image encoding format and a 
code string sequence of each of said encoded 
images is a correct sequence as said image encod- 
ing format. 

26. A moving image composing system as defined in 
claim 24, wherein, when receiving only one 
encoded image, said image composing means 
does not produce a multi-image combined image, 
but changes said information value defined as a 
variable-length code in said encoded image and 
outputs an encoded image after the change of said 
information value. 

27. A moving image composing system as defined in 
claim 24. wherein, in changing said information 
value defined as a variable-length code, said image 
composing means, when a basic operation unit 
length of said image composing means is taken to 
be N bits, inserts a bit code string corresponding an 
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arbitrary multiple of N into said extended code 
string to adjust a bit string corresponding to a varia- 
tion of the information code length occurring due to 
the change of said information value. 

5 

28. A moving image composing system as defined in 
claim 24, wherein said operation unit length is one 
of 4 bits, 8 bits forming 1 byte, 16 bits, 32 bits and 
64 bits which are operation unit lengths of a central 

10 processing unit of a general-purpose computer, 
respectively. 

29. A moving image composing system as defined in 
claim 24, further coirprising delay means for out- 

15 putting inputted data after said inputted data tempo- 
rarily or long undergoes processing including an 
inputted data accumulation and an inputted data 
retention, wherein said image composing means 
receives said encoded image produced through 

so said delay means. 

30. A moving image composing system as defined in 
claim 29, wherein a process in which said delay 
means is not placed exists in a process on one of 

25 an input side of said image encoding means, a 
process between said image encoding means and 
said image composing means and both the proc- 
esses, and said image composing means receives 
at least one encoded image produced without a sin- 

30 gle passage through said delay means. 

31. A moving image composing system as defined in 
claim 24, further comprising a layout table for, when 
a reproduced image of a multi-image combined 

35 image to be outputted is divided into one or more 
areas, retaining, as multi-image layout information, 
a construction of said areas, with said layout table 
retaining, as said multi-image layout information, at 
least positional information on disposition of each of 

40 said areas in a multi-image combined image and 
size information of each of said areas, and with said 
image composing means changing an information 
value defined as a variable-length code and an 
information value defined as a fixed-length code to 

45 values retained in said layout table in producing 
said multi-image combined image stream. 

32. A moving image composing system as defined in 
claim 31, wherein said encoded images are distin- 

50 guishable by single identifiers, and said encoded 
images having arbitrary identifiers are obtainable 
through the use of said identifiers, and said layout 
table further retains said identifiers of said encoded 
images to be displayed in said areas as area image 

55 information in said multi-layout information, and 
said image composing means refers to said area 
image information to select said encoded image to 
be displayed in said area from the inputted images 
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or to receive it as one of the inputted encoded encoding means, 

images to change said information value of said 
encoded image. 

33. A moving image composing system as defined in s 
claim 31, further comprising layout table changing 
means for changing a value of said multi-image lay- 
out information in said layout table, with said layout 
table changing means changing said value of said 
multi-image layout information at an arbitrary tim- 10 
ing, and with said image composing means refers 

to said multi-image layout information value 
changed to dynamically change a multi-image lay- 
out of said multi-image combined image to be out- 
putted. 15 

34. A moving image composing system as defined in 
claim 24, wherein said image encoding format is for 
a static image, and said encoded image is a static 
image. 20 

35. A moving image composing system as defined in 
claim 24, wherein said image encoding format 
relates to moving images, and said encoded image 

is a moving image. 25 

36. A moving image composing system comprising: 

first and second image encoding means each 
for encoding an inputted image on the basis of 30 
an image encoding format to output an 
encoded image; and 

image composing means for receiving at least, 
one encoded image outputted from said first ; 
image encoding means and at least one 3s 
encoded image outputted from said second 
image encoding means to produce a multi- 
image combined image, 
wherein said first image encoding means out- 
puts an extended code string, defined as a 40 
code string which does not affect an image in 
said image encoding format, into said encoded 
image in advance in consideration with a case 
that a change of an information value defined 
as a variable-length code in said image encod- 45 
ing format takes place, and said second image 
encoding means does not output said 
extended code string into said encoded image, 
and 

said image composing means changes said so 
information value to a different value through 
the use of said extended coded string for said 
encoded image outputted from said first image 
encoding means, and further, changes said 
information value to a different value by output- ss 
ting said extended code string into said multi- 
image combined image for said encoded 
image outputted from said second image 
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